Hi, thanks for having me. It's very humbling to be among such a, such aesthetes and to be preceded by such aesthetes in particular. I'm still sort of reeling from that bitter taste and the sort of formaldehyde jar. But could we have the, All right, excellent. So I'm going to talk to you about some work on synesthesia and food, and using computers to compute synesthesia. My dissertation work is on taste, on modeling people's points of view and their cultural tastes by looking at things like their blog, their social network profile, and my hypothesis is a psychoanalytic one.
The idea is that if you like A and you like B, there's some fundamental correlation between A and B that you don't know, it's in our unconscious. And the same thing with food. A fun hypothesis I like to provoke people with is that anytime you're going through the day, you're looking at artwork, you're talking with people, anytime you're making any choices, that all impinges upon food, and it always suggests an underlying desire or... sort of an underlying longing or need for food.
And so the idea behind the synesthetic cookbook is that any idea, any emotion, color, sound, can suggest food. And so anything actually can suggest anything. So, I'm going to tell you about two particular things. The first is, what's for dinner? I want a machine to help me figure out what I should make for dinner, in terms of recommending recipes. And the question is, can a machine imagine how a recipe is going to taste? And the answer is, yes it can, if you tell it certain things.
A recipe has ingredients, procedures, context attached to it. And if you can tell it a little bit about each ingredient, each procedure, what the kitchen chemistry is, what the flavors and evocations are, then it can imagine how something will taste. And the power of that is that, not that it can replace aesthetes, it can't, but what it can do is, for 98% of the population, that sort of revere recipes is these monolithic units that cannot be changed at all.
What it does is, it gives them an opportunity to you know, behave, emulate the behavior of aesthetes, in that they can enter some keywords that they feel like, "I want something yummy, primal," and it will pop up some recipe suggestions because the computer is doing the imagining beforehand. And... So the second question is, can a machine behave as an aesthete and actually create recipes? And this requires some sort of data mining technology that's applied to a corpus of recipes, 160,000 recipes.
What's nice is that a recipe is basically a meme. Someone said, "These things taste good together under this context." And what I did was had the machine deconstruct a recipe and says, so that instead of regarding a recipe as something monolithic, it says, "Okay, well, these things together make sense. Okay, well, these ingredients together make sense. Well, what's the underlying fabric of things that make sense? How should recipes' ingredients connect to each other and influences connect to each other?" So I'm going to tell you first about how we've taught machines to imagine the taste of food.
And so there's a website called Open Mind Thought for Food, which in my lab, we've done a whole vein of work on common sense computing, as in having people type into the web things that everyone knows but no one's going to say because it's so obvious. And the machines don't know what a sort of a lemon tastes like. And so this interface is to acquire these sentences about food, like lemons are yellow, lemons have zest, lemons taste sour.
And people can type in these facts, like when eaten, ripe avocado usually has the palate texture, let's say, mushy. And you take thousands of these, tens of thousands, hundreds of thousands of these facts, and you're really getting out of these are the connections. You're getting an annotation that says, "Ripe avocado has these flavors." So this is what I call gustatory common sense, and it was acquired through this website. And we have about 54,000 descriptions of ingredients, cuisines, cooking procedures, just annotations which come to mind.
This is not, the people who are annotating this aren't, you know, aesthetes and critics. What they are are everyday people, but there is something emergent about their common sense. They share sensibility. And so, also to make aesthetic associations that are a bit more advanced, I basically had a computer algorithm read two volumes which I are very due to my heart the Oxford Companion to Food and the Food Lover's Companion and these have dictionary entries about Everything sort of under the Sun and what's nice is that you can mine?
Associations out of those so that anytime you think of let's say Asiago you think of something else, you know You think of feta you think of Greek and out of these associations we mined out 980,000 aesthemes which is an association strong association between two things and also 100,000 context because some things, your interpretation changes as the context changes, like Tim's experiment with the jar and the labeling between body odor and what was the other one, prosciutto?
Yeah, yeah. You got me on that one. So here are some of the synesthetic keywords that the system would be able to handle. And they're sort of very arbitrary. Like you can map music and sounds and smells and everything and artwork even into taste space. And these are statistically learned associations. And the idea is that one keyword is not going to give you the recommendation of what's for dinner, but it's the intersection of all of those and the latent connections that we don't consciously think about all the time that's really interesting.
Okay. So here actually is the interface, which is created with the help of a collaborator, Matthew Hockenberry. And it's basically a plate, and you type keywords into the top, like chicken, onion. You get these recipe results which fit that criteria. And after looking at some of the things, you might type some more, like spicy. So now you have spicy recipes coming up. And you see, oh, buffalo wings, I don't like wings. So type no wings. And as you see here, there's little critters here.
They're actually your family members. And they can program their own taste buds. And as you browse, you're going to get their feedback. So, here's that detail. Sally, so she can actually do this on IM. This is a nice interface. So I want rich, moist, spiced, I must have... "No nuts, not healthy, niacin." So it takes a few hundred thousand of these keywords. The idea is to accept everything, every word that can be spoken. And so the mom's browsing for something to cook, you know, "Niacin, niacin, I want niacin." And so these are picky critics.
But that's a point of view. That's really what it is. It's a set of contexts held together loosely and reacting against everything. So the second part is teaching machines to compose foods. which is very provocative for this crowd. So, recipe deconstruction. I happened upon sort of a corpus or a collection of recipes, 160,000 of them, and being that my background is in computational linguistics and artificial intelligence, I parsed these recipes. It's actually much easier to parse than just plain English because everything is laid out for you.
it's easy to normalize. And what I do is, rather than regarding each recipe as a thing to be revered, I say, "Okay, well, every time the machine sees a recipe, it remembers that these ingredients go well together, these procedures go well together in this order." And after looking at all 160,000 recipes, it's created the deconstruction. It takes... 24,000 unique ingredients mined from this corpus, and it groups it into 5,300 families of ingredients. Some are very easy to expect, like minced garlic goes with garlic cloves, but some are sort of very far off and far-fetched.
And also, the same is done with preparation techniques. That's collapsed into families. And it also associates ingredients and procedures with the cuisine that it could be indigenous to or make sense with respect to. So, statistically, you get a very interesting view of food. And what's nice is that I feel like sort of an archaeologist, and I'm excavating from these recipes all the good ideas had through, you know, recent history. And so these are the top 20 ingredients in this corpus.
You can see that it has a bias toward Americana and also toward baking. Unfortunately, there's not been able to parse foreign language recipes quite so readily. So, you have this frequency chart, and it's interesting because just like language, the distribution of ingredients is exponential. As in, like, the most frequent, like the word "the" is used so many more times than, like, you know, the third or fourth most frequent. So let's say you look at Italian cuisine, and these are the relative significances of these ingredients in Italian cuisine, and you can see that it sort of, you know, not really, but it sort of maps the contour, the general contour, where salt and sugar are the most prevalent.
But then you look at another profile, let's say Chinese cuisine, and you can see it's completely different. And what's interesting is that if you could take every cuisine and you could reduce it to this list of frequency bars and histograms, that would give you an incredible amount of information already about what the taste profile of that cuisine is. So I said, you know, the computer looks at every recipe. Every time it sees two ingredients together, it says, okay, I'll remember that.
And every time I see those two ingredients repeatedly, I'll reinforce that association. So what's nice here is that this is a self-organizing map of correlations between things. So I paused this for a second to show you that... So every link means that these things go well together. They complement each other. Like, you know, canola oil, minced ginger, baking soda. But what's nice is in a self-organizing map in a network is that when you... You say these two things go together well, these two things go together well, and you put them into a map, it organizes and it creates cliques.
And you can actually see that this is sort of like the baking clique, or, you know, French bordering up over here. And if you keep going on this side, we can see another clique. Over here we have some spices, right? And we have more dry spices over here. And I think this goes a little bit further. Yeah, so you have on this side bordering Indian or Mediterranean cooking on this side. So that's an example of how, if you mine things like in this way, you get a sense of emergence, which is something that you didn't know could be had, just that there are fundamental relationships between ingredients.
Okay. So here's another one. And so I like to provoke you with a thought that authenticity is really an aesthetic closure. It says these things go together well. You leave out these things which would vulgarize this thing. And these things are, they make sense with respect to each other. They form a clique. Okay. And so what this visualization is showing is that, so you have this middle node called Who Am I? And it's just collecting things.
And I've typed three ingredients to collect around Who Am I? So it's like a guessing game. I'm thinking of a cuisine. It has cumin, turmeric, bay leaves. And all the yellow notes here are the other things it predicted that would be in the same recipe. It predicted chili powder, you know, oregano, carrots, paprika. So it's got the general sort of region, right? And then using a similar approach by comparing cuisines, like, oh, this ingredient is in this cuisine, but it's not in this cuisine.
we extracted some essences so here are actual top results for essence of Thai, essence of enchilada and you can actually get essence of any word just by looking at all the ingredients which collect around that word So here actually is a recipe composer. It's called Gulp Fiction. And the idea is that, so I typed the word blue, and this is synesthetic because it's got all of it. And it actually, it's writing the recipe as I type it.
Blue, it's got blue cheese, mayonnaise, sour cream, you know, white wine vinegar, Worcestershire sauce. So it seems like it's looking for acidity, but it's not instructed that explicitly. And it also tries to invent a little procedure for cooking it, which is abstract at best. So now you type another word, blue and sudden, and now what it's done is it's intersected blue and sudden. So what happened here? Well, you know, you got lime, pineapple, maybe some more pleasant things, because sudden could be pleasant.
You got some broccoli, um... And the procedures change. And then you type the final word here, "salsa." And so you have -- all right, so now you have something which is kind of salsa-y. The little cheat here is that the way the recipes are named, generally the last word is the most salient, so you have to give it the most weight. So you have lime juice. You have a very interesting salsa because it has blue cheese.
But, you know, in the context of this crowd, this is art and not garbage, right? So here's another one. We have, and this is, we're going completely off the deep end here. Mediterranean, which, you know, this is a typical profile. I got balsamic vinegar, olive oil, aubergine, etc. And also, the aubergine and the eggplants are, you know, obviously, just like the erycoverian green beans, have completely different contexts, just because they contain sort of a lot of contextual information, even though some people would argue that the taste profiles are pretty similar.
So we have Mediterranean, and then we're going to disrupt that, and you're going to throw in caveman. So, what it's done really is that we're making beef short ribs, because apparently that's very primal and courgettes, but it's Mediterranean because it's got feta and it's got capers and balsamic vinaigrette. So, by the way, if any of you have restaurants, would like this cooked, be happy to entertain this. I had the pleasure of doing a couple of these food performances on the invitation of some folks.
And I think one group, there were artists that were invited to this dinner, and they gave words such as "misunderstood," "gravel," whatnot. And the strange thing was the intersection of all those was like a very strange black bean porridge. It was very dark. So here are some recipes around paella. What I wanted to tell you is that, you know, we're all aesthetes here. We're in the 2%. And in 98%, what would it mean to someone if you gave them a recipe for paella and then you allow them to modify the paella by adding other influences.
Here I've added, you know, this is Thai paella, so I've thrown in fish sauce. And by the way, the quantities are all inferred also because there's a very primitive sense of kitchen chemistry which happens in this recipe. maker and um and coconut milk um so that's sort of plausible then you have shepherd's paella which means you throw in you know mutton and uh apparently uh and uh it could be could it could work could work If it was served at one of your restaurants, it would definitely work.
Then you have something more implausible like a paella compote, which, you know, dried apricots can work. But, you know, what I think of this is it's sort of a training ground for asthites, aspiring asthites, because they can experiment with different taste profiles, mixing them together. And lastly, paella Suzette, which... I'm not sure if this is dessert mussels with blueberries and cranberries. Yeah. And what I wanted to leave you on was my discovery of the fact that, based on all these ingredients and all the combinations that these ingredients typically fall in, that only represents about 2% of the possible combinations that ingredients could be in.
So that's sort of, you know, 98% to go. That suggests a very promising future for the field of food science. So thanks very much.
Machine transcript (WhisperKit, large-v3 turbo, on this machine, 12 September 2026), corrected only where a name or term was misheard; the player’s captions carry the timings.