The small circles superimposed on this fish photo are used by the machine-learning software to automate species identification—in this case a red drum. Photo by Lisa Kellogg/VIMS.
By Edited Press Release
Researchers at the Virginia Institute of Marine Science (VIMS) are casting a wide net in search of fish photos. Their immediate goal is to use angler snapshots to train software to identify different fish species. Their ultimate goal is to put that artificial intelligence into a RecFish cell phone app, giving anglers a multi-use field guide and scientists a collaborative tool for better management of recreational species.
“We’re really excited about building out our app,” said RecFish founder Lisa Kellogg, a senior research scientist at VIMS. “Once finished, all you’ll need to do is open the app and point your cell phone camera at a fish and it will instantly tell you the species, size, and approximate weight, and let you record that info to your logbook with the touch of a button.” The app will also instantly provide information on whether the fish is legal to keep based on the date, location, and local fishing regulations.
But before that can happen, the app development team must first train machine-learning software to recognize different fish species, a process that requires lots and lots of photos. “Right now,” said Kellogg, “we’re working on training models to identify fish and that requires at least 5,000 photos per species. The more photos we have, the more accurate the models will be.”
To help collect these images, the team has developed a website at recfish.org that makes it easy for anglers to upload their photos. Kellogg says, “We want people to look through their photos and upload anything that has a fish in it, so that we can use them as part of the model training.” Their initial focus is on fishes of the Chesapeake Bay, but their long-term plans include expanding coverage to the U.S. East Coast and beyond.
The team is particularly interested in photos of less common species and species that people rarely photograph. “There are 270 resident species in Chesapeake Bay,” said VIMS fisheries professor and RecFish team member Dr. Eric Hilton, “from common fishing targets such as striped bass, croakers, and flounder to more unfamiliar species such as stargazers and sea robins.”
“In reviewing the photos we’ve collected so far,” said Kellogg, “we’ve found that everyone takes photos of that huge striped bass they caught but far fewer take photos of their toadfish.” Particularly surprising has been the difficulty in getting photos of even some of the Bay’s most commonly caught species.
“Spot are the most commonly caught species in Maryland and Virginia but we still don’t have enough photos to model the species,” says Kellogg. “We’re hoping to get the word out that we need photos of all of the fish you catch, not just the ones you’re most proud of catching.” Despite the low number of photos they are currently working with, initial modeling is going well, with some species models already more than 90% accurate.
Kellogg and her project team appreciate that participating anglers may not want to share the whereabouts of their secret fishing spot. “RecFish will let you choose whether to share your location data with scientists and resource managers,” she said. “You’ll be able to share your exact location, which has the greatest value for scientists, but if you’re uncomfortable with that, you can submit your data just by tributary segment—upper, middle, or lower York River, for instance—or with no geo-reference whatsoever.
“Even if some anglers never contribute their catch data,” she added, “the app will contribute to sustainable fisheries management by helping them identify their catch and providing up-to-date fishing regulations.”
Hilton notes that the app will also aid with basic fisheries science. “Anglers collect more data than any fisheries biologist ever could,” he said. “Because they’re covering such a large area, including places that aren’t sampled scientifically.” Shared photos will also help document new or unusual occurrences of species within the Bay.
In addition to Kellogg and Hilton, the app development team includes VIMS researchers Sarah Muffelman and Jennifer Dreyer; Dharmesh Trivedi, Harshil Shah, and their staff at the machine and deep learning firm DXFactor; and Rob Quartel, Innovation Advisor to VIMS Dean & Director Dr. John Wells.
App development was jumpstarted by support from the Dean & Director’s Innovation Fund at VIMS, established in 2018 by the Joan and Morgan Massey Foundation and the Nunnally Charitable Trust. Kellogg was one of two VIMS researchers to receive the fund’s inaugural award in 2019, and she received it again in 2020 for further app development.
“The goal of the Innovation fund is to provide support for research and education activities that foster innovation and economic productivity at VIMS,” said Wells. “What Lisa and her team are doing with their RecFish app is a great example of the kinds of projects we are looking to foster.”
“The VIMS Innovation Fund encouraged us to think outside the box and come up with this concept in the first place,” said Kellogg. The National Fish and Wildlife Foundation has provided additional funding for app development, and a wide range of management agencies and NGOs have expressed support. Kellogg expects the app to be available to the public by the end of 2021.
The RecFish developers want to hear from anglers about other looked-for features, with a form on the RecFish website for user feedback. “We want anglers to tell us what else they’d like to see in a recreational fishing app,” said Kellogg. “Do they want to know whether the fish is edible, or about health hazards, like a high mercury level? Do they want more information about the fish—its diet, habitat, and range? Do they want to know the maximum size, the state record for that species? Would they like the app to show what you were catching last year on that day? Do they want to this to be a photo-sharing app where you can brag about your catch or compete with friends? The sky’s the limit. We’d love to hear everyone’s suggestions.”