Le Décodeur

Decoding consumer product packaging with computer vision and machine learning

Azure Computer Vision, React Native, Node.js, Python, SQL

young man applying some cream on his face

Since 1973, Protégez-Vous (French for “Protect Yourself”) has operated as a non-profit organization that performs rigorous tests, studies, and surveys on the tens of thousands of new products that are put on Quebec shelves annually. In 2020, we worked closely with them to create an AI-powered app that their team could use internally to help them maintain their database of cosmetic products. For the second half of the project, we were tasked with creating a consumer-facing app (called Le décodeur—in English, “The Decoder”) which would help people make more informed purchase decisions, just by snapping a few photos of a product’s packaging (ingredient label plus ecological logos). Read on to find out how we did it.

The Decoder - mobile application

In-app camera functionality allowing consumers to photograph/evaluate ingredient labels

The challenge

Firstly, Le décodeur required in-app camera functionality to allow consumers to photograph ingredient labels. The words would be extracted from the image and compared against Protégez-Vous’ extensive database in order to display a summary of the product’s toxicity risk level. This information would be organized based on the type of exposure to the product (for example, if it were to be absorbed through the skin or swallowed) for different groups of people (for example, babies vs. pregnant people vs. adults). Beneath the summary, relevant articles would display to further assist the consumer.

Secondly, Le décodeur needed to bridge the gap between Protégez-Vous’ database of ecological logos that are commonly found on personal care products, and the consumers who want to understand if the trustworthiness implied by the logo is actually valid. For example, certain logos indicate that a brand is meeting rigorous standards, whereas others are merely greenwashing tactics. To convey this information to consumers, Le décodeur’s in-app camera captures an image of the ecological logo and, if a match is found, the consumer is then shown a summary displaying the logo’s meaning, scope, and reliability, and how it ranks in terms of trustability using a score of 1 to 5.

The task at hand

We leveraged our skills in computer vision technology and machine learning techniques to have Le décodeur identify text and logos and match them against the Protégez-Vous’ database with a high degree of accuracy. Behind the scenes, a lot goes into extracting information from a photo. Whereas text-recognition is relatively straightforward, logo recognition is trickier because factors such as the angle a photo is taken at, under which lighting conditions, the texture or curvature of the product packaging’s surface all must be considered. In order to train Le décodeur’s algorithm to recognize subtle nuances in an ecological logo with a high level of accuracy, approximately 100 photos of the same logo are required.

However, the biggest technical hurdle we overcame wasn’t image capture itself, but configuring the context. We worked to configure Le décodeur to recognize a good result, a false positive, or a bad result (requiring a new photo to be taken by the user). During the testing period, Le décodeur was achieving an accuracy rate of 55%. At launch, that had improved to a rate of 88% accuracy, demonstrating that the way we configured the app was done with precision. But results don’t stop there. For any ecological logo that comes up as “no match,” the user will have an option to submit their photo to Le décodeur to further train the algorithm. In a similar vein, users may submit any ingredient that isn’t in the database to Protégez-Vous so that their lab techs may add it to their testing queue.

A person applying a moisturizing foam on his hands

The end result

Thanks to this great team effort that involved cross-functional collaboration with internal and external agencies, Le décodeur is now available to download for iOS and Android. Protégez-Vous has big plans to empower consumers to be better informed when it comes to decoding consumer product packaging—something that will benefit the health of Quebec families and communities as a whole—through ingredient risk factor education and providing a sense of transparency around the ecological logos that brands employ.

One of the main pages of the mobile application

What's next ...

Protégez-Vous is very excited about the future uptake of Le décodeur built in collaboration with Osedea, especially because they’ve seen similar apps in Europe have huge success. Since completing our mandate, le décodeur’s database of ecological logos continues to grow with the addition of logos for food products, electronics and household appliances. Also coming soon for Protégez-Vous is a marketing campaign, which will boost awareness about the app and how it can help Canadians make more informed buying decisions.

Le décodeur is an excellent testament to Protégez-Vous’ ability to innovate. They started over 50 years ago as a magazine, then were one of the first to move to a website, and then to a website subscription model. With over 100K subscribers today, the non-profit continues to think outside the box in order to reach new consumer demographics. It’s yet another cause that we at Osedea are proud to be involved in championing.