Added macronutrients for 100 foods

Nutrify now displays macronutrient information for ~100 foods

When you upload a photo to Nutrify, it now predicts what class of food is in the image and then displays information about that foods macronutrients.

This happens via:

image upload -> computer vision model -> returns class -> class index used to index on a database (hosted on Supabase) -> information from Supabase displayed on homepage

This took a while to happen but I had to learn all about databases (Supabase uses pure PostgreSQL).

The good thing is the app is still fast.

A prediction + query to the database is done in ~0.2-0.5 seconds.

That'll be the peak time I'd like to let it happen in, any longer and the experience won't be as fun.

See the database code in the various JavaScript scripts (I'm new to JS so these are rough):

• Constants for different food names/food data central IDs (fdc_id) live in constants.js
• The database query and HTML update for macronutrients happen via get_data.js
• The image upload and image classification happens via script.js
• The connection to Supabase happens via supabaseClient.js - this uses an anon API key, that's apparently "safe to use in public", though I'm not sure what this means
• Utility functions (all one of them) are located in utils.js

Is this how JS projects get laid out?

I have no idea.

It seems to work though.

Issues

No nutrition information for some foods

You might find that ~3/100 foods (like "egg tart") don't show nutrition information.

This is because they don't exist (at least what I've found) in the Food-Central USDA Database (where the rest of the nutrition information comes from).

Data loading is likely not optimal

I'm a newbie to databases.

The speed is how it is now because all of the database rows are loaded when you go to nutrify.app and then they are cached and queried when a prediction is made.

This may not be best practice... but it works.

Ideally, the database would be queried per image classification.

However, I tried this and it takes 1-2 seconds per query (from where I live in Australia because the Supabase database is hosted in the US), this kind of time delay is not acceptable.

Again, queries should happen within a maximum of 0.5 seconds.

The model still does not predict whether there is a food present or not

Future updates will include a food/not food model to see if there's food present in the image or not.

Next

• Add food/not food model + a message about "no food detected, please upload a photo of food"
• Make app design nicer
• Add a way to toggle different nutrients to view
• Add a way to adjust the different amount of macronutrients per food (e.g. 100g -> 50g -> 250g)
• Add logging? Logging for different predictions/images etc
• Clean up the GitHub repo... seriously, what's going on here now? Perhaps it's time for an organization?

100 food milestone (78 foods -> 100 foods)

I've just created a new model based on EfficientNetB1-Lite (the TensorFlow JS version) that now powers Nutrify and published it to the Nutrify homepage.

In short, Nutrify can now identify 100 total foods (previous was 78).

The performance for each class varies quite a bit. But this will be upgraded in future models.

Next

Now we're up to 100 foods, the next step will be making the current model better (better data) and adding metadata to the website rather than just identifying the class of a food.

In essence, the next few updates will focus on making the current system better rather than adding more foods.

So the current list of foods will likely remain fixed for the next month or so.

App updates (to come)

• Add metadata (such as calories, protein, fat etc) for the first 100 foods (see list below) and make it display when a particular food is identified.
• Add a model cascade for identifying food/not food (Nutrify should recognize a photo of food or not).
• Display the top X (maybe 3?) predictions that can be clicked on and used.

Modelling updates (to come)

• Make training reproducible.
• Make data traceable, e.g. what data is used where?
• Have training and evaluation take place with results (e.g. acc, precision, recall per class)

Updates

The previous Nutrify model could identify the following 78 foods.

Note: If there's more than one food in an image, Nutrify still only identifies one class, this will change with future models.

# Previous 78 Foods Nutrify could classify
['Apple',
  'Artichoke',
  'BBQ sauce',
  'Bacon',
  'Bagel',
  'Banana',
  'Beef',
  'Beer',
  'Blueberries',
  'Bread',
  'Broccoli',
  'Butter',
  'Cabbage',
  'Candy',
  'Cantaloupe',
  'Carrot',
  'Cheese',
  'Chicken',
  'Chicken wings',
  'Cocktail',
  'Coconut',
  'Coffee',
  'Cookie',
  'Corn chips',
  'Cream',
  'Cucumber',
  'Doughnut',
  'Egg',
  'Fish',
  'Fries',
  'Grape',
  'Guacamole',
  'Hamburger',
  'Honey',
  'Ice cream',
  'Lemon',
  'Lime',
  'Lobster',
  'Mango',
  'Milk',
  'Muffin',
  'Mushroom',
  'Olive oil',
  'Olives',
  'Onion',
  'Orange',
  'Orange juice',
  'Pancake',
  'Pasta',
  'Pastry',
  'Pear',
  'Pepper',
  'Pineapple',
  'Pizza',
  'Pomegranate',
  'Popcorn',
  'Potato',
  'Prawns',
  'Pretzel',
  'Pumpkin',
  'Radish',
  'Rice',
  'Salad',
  'Salt',
  'Sandwich',
  'Sausages',
  'Soft drink',
  'Spinach',
  'Squid',
  'Strawberries',
  'Sushi',
  'Tea',
  'Tomato',
  'Tomato sauce',
  'Waffle',
  'Watermelon',
  'Wine',
  'Zucchini']

Note: Squid and Chicken classes were removed from the above list (the Squid data was poor and Chicken as a class images are of actual chickens rather than chicken as a food).

And now with the addition of 24 more foods:

# 24 added foods to Nutrify
['Avocado',
  'Cherry',
  'Dumpling',
  'Egg tart',
  'Eggplant',
  'French fries',
  'Garlic',
  'Grapefruit',
  'Green beans',
  'Green onion',
  'Hamimelon',
  'Kiwi fruit',
  'Lettuce',
  'Meat ball',
  'Noodles',
  'Nuts',
  'Okra',
  'Papaya',
  'Peach',
  'Pie',
  'Plum',
  'Red cabbage',
  'Spring rolls',
  'Steak']

This brings the total to 100 identifiable foods (1000 is the goal for the end of the 2022).

# All current foods Nutrify can identify
{0: 'Apple',
 1: 'Artichoke',
 2: 'Avocado',
 3: 'BBQ sauce',
 4: 'Bacon',
 5: 'Bagel',
 6: 'Banana',
 7: 'Beef',
 8: 'Beer',
 9: 'Blueberries',
 10: 'Bread',
 11: 'Broccoli',
 12: 'Butter',
 13: 'Cabbage',
 14: 'Candy',
 15: 'Cantaloupe',
 16: 'Carrot',
 17: 'Cheese',
 18: 'Cherry',
 19: 'Chicken wings',
 20: 'Cocktail',
 21: 'Coconut',
 22: 'Coffee',
 23: 'Cookie',
 24: 'Corn chips',
 25: 'Cream',
 26: 'Cucumber',
 27: 'Doughnut',
 28: 'Dumpling',
 29: 'Egg',
 30: 'Egg tart',
 31: 'Eggplant',
 32: 'Fish',
 33: 'French fries',
 34: 'Fries',
 35: 'Garlic',
 36: 'Grape',
 37: 'Grapefruit',
 38: 'Green beans',
 39: 'Green onion',
 40: 'Guacamole',
 41: 'Hamburger',
 42: 'Hamimelon',
 43: 'Honey',
 44: 'Ice cream',
 45: 'Kiwi fruit',
 46: 'Lemon',
 47: 'Lettuce',
 48: 'Lime',
 49: 'Lobster',
 50: 'Mango',
 51: 'Meat ball',
 52: 'Milk',
 53: 'Muffin',
 54: 'Mushroom',
 55: 'Noodles',
 56: 'Nuts',
 57: 'Okra',
 58: 'Olive oil',
 59: 'Olives',
 60: 'Onion',
 61: 'Orange',
 62: 'Orange juice',
 63: 'Pancake',
 64: 'Papaya',
 65: 'Pasta',
 66: 'Pastry',
 67: 'Peach',
 68: 'Pear',
 69: 'Pepper',
 70: 'Pie',
 71: 'Pineapple',
 72: 'Pizza',
 73: 'Plum',
 74: 'Pomegranate',
 75: 'Popcorn',
 76: 'Potato',
 77: 'Prawns',
 78: 'Pretzel',
 79: 'Pumpkin',
 80: 'Radish',
 81: 'Red cabbage',
 82: 'Rice',
 83: 'Salad',
 84: 'Salt',
 85: 'Sandwich',
 86: 'Sausages',
 87: 'Soft drink',
 88: 'Spinach',
 89: 'Spring rolls',
 90: 'Steak',
 91: 'Strawberries',
 92: 'Sushi',
 93: 'Tea',
 94: 'Tomato',
 95: 'Tomato sauce',
 96: 'Waffle',
 97: 'Watermelon',
 98: 'Wine',
 99: 'Zucchini'}

More class-based results to come soon.

Here's a visual update of the new classes (I've just noticed the duplicate of Fries and French Fries), this should be ironed out in the next update.