Webinar Do science art and technology spice up future food experiences 10/12/2021

Show video

...professor of food development at the University of Turku and welcome you all to our webinar   'Do science art and technology  spice up our future food experiences.'   This question was raised by a group of researchers  and artists from different disciplines of food   science, technology and art, as we were discussing  the future visions of food experience and dining.  Technology is an increasingly diverse part of our  lives and the dining experience is no exception.

However the essence of eating remains unchanged  when we eat we share a common experience and we   are together present in a unique moment. The moment of dining is the core element of our community and this is also the case in the technological world. We raised the question how to harness technology  to serve that age-old meaning of communal dining. 

Could the combination of technology and art  provide an experience of a shared meal with   a friend even if we were sitting alone in  our kitchen on the other side of the world. How could we enhance the feeling of communality  and commensality by combination of technology and art?  Maybe transferring the movement of the  fellow diner's spoon into shared soundscape   and then further into the tactile sensation of  your neighbor's body? And we should also learn how do these new technologies shape the sensory and emotional experience of eating and dining. For example; how does cheese and wine  flavored with virtual technology taste like? And during the autumn of 2021 our group has  been working on these new spices of   future food experiences, focusing on communality of dining and its importance in the digital world. And in this December webinar we sum up our results so far. We have chosen fermentation for our artistic forum   and virtual and mixed reality dining examples,  because fermentation is the technology, that   humans have used to produce foodstuffs and varieties since the neolithic age. And we want to acknowledge this ancient technology  and connect it with the most recent ones.

You can see here the program of the webinar.   The webinar is divided into two sessions which  both are divided into presentations on science   and then presentation on art and technology.  And after each session there will be time for   questions and discussions and for the  webinar participants the chat is open   all the time for questions and comments, and they will be shared during the discussion.   This first session is entitled 'From neolithic  times to virtual and mixed reality dining' and it takes us from fermentation and  the world of microbes via human sensory   experience into the possibilities  of virtual and mixed reality dining.  

And by these words I welcome our first  speaker, professor Lucrecia Delfederico from   National University of Quilmes, Argentina, to take  us to the world of microbes, Lucrecia, please. Thank you very much for having me here today.   We are talking about fermented foods, the other domestication, wine and cheese. The domestication of plants and animals during  the neolithic allow humans to organize certain societies  and develop technological innovations  for the treatment and conservation of food, cooking, drying, salting, smoking and of course fermentation.  The fermentation provided new foods with a wide   range of flavors smells and textures, and in many  cases more digestible than starting materials. 

There is evidence of great wine and  viniculture, that cames from georgia, in   the south caucasus regions, dated  around 6 000 years before Christ.   It is the earliest biomolecular archaeological  evidence for great wine and viniculture,   as we know it in the west, and it is crucial to  the later history of wine in Europe and the rest of the world.  The oldest known evidence of cheese  making are pieces of pottery full of holes   discovered in Poland, of about seven thousand  years, with remains of dairy fats in them.   The first farmers have used this object  to separate the curd from the liquid whey. Simultaneously and most likely by  chance the domestication of bacteria,   yeast and molds also took place, mainly through  artisanal food production practices, such as the   cereal reinoculation of new foods with materials from previous food.

These practices led to the selection of microbial communities adapted to the characteristics of the food to be fermented. Neolithic humans so harnessed the  metabolic capabilities of microbes   while they were trying to improve the  digestibility, palatability and self-life   of their newly abundant food. So fermentation is  one of the oldest biotechnologies applied to meat,   fruit meat and grains to transform  and preserve them by microbes.   Despite the fact that wine and cheese are  such different food matrices they both depend   on interaction between microbial  communities for their development.   The structure or compositions of these  microbial communities differ between both   fermented products, but a group of micro-organisms  plays a fundamental role, lactic acid bacteria.

The basic manufacturing process is  the same for all cheese, actually   it's a dehydration process of  milk. Rennet and starter cultures are added to the milk, the milk coagulates,  the coagulum is cut and the whey drained.   The remaining curd is pressed in the form and allowed to ripen.

For fresh cheeses the curd is cut and drained, but not  pressed, and they are not allowed to ripen.  What actually differentiate cheeses and strongly  influences the final products, is the use of   different starter cultures of micro-organisms.  The presence or absence of mold, and the method   of weight separation and ripening methodology.

Also variations in size, shape and all other ingredients added lead to the local   production peculiarities and  almost infinite variety of cheeses.  According to their role the microbiota involved in the process of cheese manufacture and ripening  are starting lactic acid bacteria, such as mesophilic Lactococci or Leuconostoc,   thermophilic strains of Streptococcus thermophilus,  which are typically used for the production   of semi hard and hard cheeses, and adventitious non-starter lactic acid bacteria, mainly Lactobacilli.  The starters are responsible for the acid development during cheese production  and contribute to the initial ripening process. The non-starters have been shown to  play a contradictory role during ripening by enhancing flavor and sometimes   deteriorating cheese quality. Flavor develops in  cheese by the combining metabolic activities of   microbial communities on milk fat,  milk proteins and milk carbohydrates.

And the main kind of carbohydrate is the lactose.   The depletion of lactose in cheese curd  is essential to avoid the development   of undecidable secondary micro-organisms  which will negatively affect cheese quality. The citrate present in milk constitute a secondary  energy source for some LAB (lactose acid bacteria) and the co-metabolism of lactose and citrate leads to production of carbon dioxide   and aroma compounds such as the acetylene. The major LAB metabolic pathways involved in cheese flavor formation are metabolism of lactose via glycolysis,  metabolism of citrate and metabolism of the fat.  

For lipolysis and proteolysis and the  subsequent metabolism of amino acids.  In some cheeses secondary microbiota is added to enhance proteolytic activity during ripening.  They are molds like Penicillium camemberty, Penicillium roqueforti and Geotrichum candidum,  and also bacteria  as Brevibacterium and Propionibacterium. Moving on to the wine, it consists of around 86 percent water, 15 percent alcohol and 1 percent secondary metabolites. 

This complex mixture of chemical compounds define the wine appearance, the wine color, aroma, flavor and mouth-feel properties. Sensorially active compounds originate  from the grapes, from microbes which come   from the soil of the vineyard and from  the winery and, when it is used, from oak. Grapes and grape must naturally contain mixture of microbes. During fermentation there is a  sequential development of some yeast and bacteria. And once anaerobic conditions are established in   the vat or tank, nutrients become depleted and the  concentration of ethanol increases. Such conditions favor Saccharomyces cerevisiae with the most efficient fermentative metabolism that are able to withstand low PH, high sulfide and high ethanol environments.

A widely used practice is to inoculate the grape must with fast fermented strains   to ensure a rapid and reliable fermentation.  However by doing that, the desirable yeasts originating from the grapes, have limited opportunity to contribute to the complexity of wine  Thus to preserve the regional characteristics of the wine  some wine makers prefer to conduct spontaneous fermentations   comprising mixed and sequential dominance of  indigenous non-Saccharomyces and Saccharomyces yeasts.  A wine yeast converts grain nutrients into ethanol,  carbon dioxide and secondary metabolites so it's glycolytic metabolism, which impact  aroma, that can enhance the quality of wine.   Some grape derived compounds, such as acids,  including L-malic acid, and polyphenols,   remain largely untouched by the yeast cells  while other compounds are fully generated.  A secondary process that can improve the quality  of many red wines and some white and sparkling wines,  is malolactic fermentation, carried  out by lactic acid bacteria during or after   alcoholic fermentation. Malolactic fermentation  is in fact a decarboxylation process, in which   malic acid is converted into carbon dioxide and  lactic acid, resulting in the acidification, flavor modification and microbial stability of wine.  Oenococcus oeni and Lactiplantibacillus plantarum  

are the main responsibles for this process,  since they are the best adapted species to the   harsh conditions of wine. The biosynthesis of  aroma compounds during malolactic fermentation   includes the activity of various enzymes  present in LAB, such as beta-glucosidase.   Malolactic fermentation improves wine  flavor by reducing herbaceous notes   due to six carbon alcohol content and enhance  the fruity aroma increasing ester contents.   Also malolactic fermentation often occurs  spontaneously by action of lactic acid bacteria   from grapes and cellar, it implies fruits such as  consumption of residual sugar and formation of   undesirable biogenic amines. The use of  indigenous starter cultures best adapted to  

a specific wine production area is recommended  to maintain the wine regional characteristics. In the wine industry the growing demand  for distinctive regional flavors has then   led to the concept of the one defining that the  set of physical and biological characteristics as   well as local viticulture and winemaking techniques, which together determine the sensory   attributes of the wine. There is evidence that  microbial activity is associated with wine terroir.  Nowadays we know that the bacterial and  fungal communities of wine are shaped by the   grape variety, by geographical and climatic  factors, and by wine resources.   Thanks to next generation sequencing techniques, that  have been used to monitoring seasonal change in   the microbiota present in vineyards and wine, this  technique allow to prove that the consortium of   fungi and bacteria involved in wine fermentation correlates with the chemical composition of the finished wine.  And the last word can be applied to wine, to cheese and other fermented food   because it is a collaboration. The collaboration between nature and culture express  

the characteristics of the region and the history  and culture of the people who make it. Thank you. Thank you very much Lucrecia for introducing  us to these fellow microbes, that make the food delicious food for us. And now we go into the deliciousness   from here, and I invite professor Mari  Sandell from University of Helsinki to   share her experience on multi-sensory experience and especially cross modality. Mari, please. Thank you.

So hello everyone. I'm professor  in sensory food science and then also   in sensory perception, both at the University of  Helsinki and University of Turku, and I've been dealing,   my sensors are like human senses,  so I have been dealing with my group   called Senses and Food in this area already quite many years, and it's very very fascinating.   And I try to activate you very different ways now  in this next 10 minutes that I do have my speak.   When we are talking about multi-sensory   perception, this is how I understand it, so   that we do have five senses, which are activated  different ways, and the objects or the stimuli,   stimulus, it may be food, it may be both,  in case of food, it may be also eating environments  but it may be any other  object that our senses are perceiving.   And we may have five senses or then we we may  have less, so it depends on the person, and this   is very individual. Our senses are working very  different way, but in many cases, especially in food   there are several senses that are activated  together and they are really absorbing    the things from the food. The perception part  is pretty complex, and for different senses  

there are different ways how they are processing  the information from the object, and how they   are processing the information in our brains, so  it is very complex. And maybe more complex is   the way that how our experiences  are really result. And maybe in case of food quite often the wish is that food will create  such kind of woe effect or at least the   pleasant effect something that I really remember for a long time as a very good experience.  But it may happen that the experience is also negative or it it may be even neutral.  and it depends on the person as well and this is very complex  there are very different kind of emotional factors that are also activated  and we can create also the emotional profiles for different experiences.  

Now I'm activating you with one photo, which is  one of my favorites, some people scare it   some people think that it's very ugly, some  people think that that's something that   I would like to try, so let's see that's how  you are now perceiving this visual stimuli.   Think about this nice plate of vegetables. There  are different kind of colors you can see,   very different kind of plants and it's very  healthy, it looks that it's something that   we are supposed to eat. Some people may see here a  happy face, some people may see here that there is  

something very weird going on. The question is that  to eat or not to eat, and people are reacting   in very different ways to this photo. It would be nice  to discuss with you about your reactions as well.   Flavor is something - the word flavor is missing  here - here i'm explaining that how the flavors   are actually formed, so the flavor which in  many cases is confused with the name of taste   it is actually the combination of retronasal  smell, taste and chemesthesis. I have been studying   strawberry flavor quite a lot, and if we  think about it from those  factors that are contributing to flavor, there  are different sugars and different acids and very   different kind of volatile aroma compounds and  also polyphenolic compounds, and there are   different kind of smells and and tastes and  chemesthesis when we do have the strawberry   in our mouth, so flavor is truly the  multi-sensory perception, it's an experience.  

We have studied, in the project funded by Academy of Finland,   in my group the sensory perception with Finns,  and this is an example where we ask people to look   at different kind of colored bottles and we ask  them different kind of questions, for example put   them to order, how sweet you think that the samples  are, how sour they are, and this publication is   a demonstration about the fact, one of  those publications, that how by our sense of   sight and different kind of colors are activating  also the taste in our mind. And one point in   this publication is also that those reactions are  different, so we are not similar as human beings. Another publication, which was  published last year, was dealing with   food consumption and salad buffet,  and we applied here the multisensory environment  so we had the control environment, control space, and then we   did have a forest kind of environment,  where people were selecting and eating the food.   And actually we didn't see that great impact  on the environment in this food choice and the   buffet choice, you can see the buffet there in  the photo, there were 14 different kind of food   items that people were allowed to select,  but what was interesting that when we asked   also the emotional profiles for these different conditions   actually in the case of the forest landscape,  people felt that they were more happy while they   were also eating and selecting and eating the  food, so even more complex combination   of both the food and then the environment.  Then into the end of this talk I selected    several publications, that have been published  this year, concerning the multi-sensory perception   and different kind of food items. For  example there is one publication that's

study topic was beer, very fascinating, quite different,  by several researchers around the world   and then another  publication from Denmark   how the visual attention and sounds are contributing to the consumer food choice.  Then cookies, and dining-out during corona conditions.   These are all public publications, so just  go ahead and read more about them. I'm not going  

to the details, but showing you that what kind of  different things are published during this year.   Sense of smell, so olfactory censory, and then  the audio visual integration, it's very   fascinating area for researchers and there is  one example of those. Then one example of the    odor, taste and texture interaction, and how this can be  connected to the health issues, such as overweight.   Both the children and adults, they have been  studied because children and adults may also   differ a little bit, and then let's remember  that there are also individual differences.   Then the taste and smell, and how they  are contributing to the combined flavor, which   was published just a couple of months ago. Then  the third slide still about the visual and taste,   their connections. Sweetness is always pretty, pretty interesting. How to enhance the sweetness without adding sugar  

And here we're also a little bit going to the virtual reality world, that we will hear in the next presentations much more,  so I'm not going into details in that.  These are very nice examples, all of these,  that what all kind of activity   we have in the area of the science concerning the  multi-sensory perception and also related to food.   That was shortly my presentation, my 10 minutes.  Thank you for your multi-sensory attention and   I wish I activated you also with this photo from  the real life, a lot of food, a lot of colors,   and I wish that there's at least  something for everyone. Thank you. Thank you very much again for your   inspiring presentation on  multi-sensory perception in a sort of,   in this traditional world. It was interesting to hear that multi-sensory   perception and cross-modality take  place also in the real world, and now we   move on to doctor Jean-Christophe Sakdavong's  presentation from University of Toulouse   and we hear state of VR and AR and so forth. Please  go on.

Thank you. So I have a PhD in Computer Science, but I'm researching cognitive psychology  and ergonomics, so it's a specific profile.  And then we try to make a fast state of  art part of what we can do with virtual reality,   augmented reality, mixed reality, for future food  experiences. So I will start first by speaking about mixed reality. Then the factors which have been identified.  I will do as my colleague, you'll  see a little spectrum of publications and the   researches which have been done. Unfortunately the  only one she selected is the one I was not using,  

but I know it and we'll see a bit what we  can do in the future. So first mixed reality.  About, first of all virtual reality, I think  everyone knows what it is now. It's usually a   computer generated simulation, it's in three  dimensions and you have most of the time a headset.  

You can include in virtual reality use case  watching 360 degrees videos, and it's quite   used in the sensory researches, so that's why I  speak about it. Augmented reality, at the beginning   it was simply adding information to the real world,  the same as your car display the speed you   are driving for example. This was the  beginning. After we have started to add 3D   virtual objects, and what we call currently mixed  reality, we could call it augmented reality, but   with evolution of technologies people have  tarted to speak about mixed reality, which   is a mixed and an interaction between  virtual reality and augmented reality.   And it has come with the most  modern technologies as you can see.   Here for example with Microsoft Hololens, where  you have glasses, which are adding informations.  

But you can also have it in a simple smartphone  using the pass-through camera and watching the   real world with other things, and also with  virtual reality, when we have a pass-through camera. So, if I try to use a reference, I can speak about the virtuality continuum.   and if you see how the things are coming  virtual, you can start by the real environment   and the more you become virtual, it's like this, so  you are in reality, you are with a virtual headset   and you can augment the reality with information,  three objects. You can also see it in the other   direction by augmenting the virtuality by   the real world. So for example a virtual world   with a small window like this one   and it will be a kind of augmented virtuality.  

This is the usual use, and usually people see this  with only two senses, the visual and the audio.   So you have a real environment with everything but  they take care more of visual and audio elements   and you add visual and audio elements. The  first augmented reality and the beginning of   mixed reality it was the museums with these headsets,  which are telling you the story without the guide,   but it can be also visual. So you see it's  about visual and audio. In our case we are a   bit more interested in real physical elements or  reality, it could be food or dreams. So we can augment   the real world with our virtual elements  or we can augment the virtual world with   food elements, with drinks. And it is more   what are doing in one side, and the   other side it can be like this for example.  Here you can change the appearance of food.  

Or here you can make a virtual world, where  you are navigating and grab some fruits   in the trees, for example. And this  is a bit where we are going, what   is interesting to me at least. And so now I will speak about more cognitive  psychology factors that I have studied   and which I think are important for sensory elements. The first one   is immersion. And immersion is, to understand  it, I mean the proper way, the right way,   which is as an objective factor, which means  it's not depending on the person. It's depending   on what you are using; a big headset, a small  phone - this is how you can describe   the degree of immersion - and a speaker, but I can  hear everyone, or a noise cancellation headset,   it will change my degree of immersion. And so we  can do it well, it's only depending on technology . 

And many researchers are speaking about it,  mixing it up with the feeling of presence.   The feeling of presence you can describe it  as a sense of being here. If you feel you are in   this world and it's not depending on how much  the technology is good, it's a bit more complex   and it's a more individual. It's the same as  for food enjoyment, the emotions that we can get.   And the most recognized researcher  in cognitive psychology about it, Lee,   has described with three main dimensions, spacial  presence, the social one and the self one. For us the social is not so important here, except if you  want to enjoy food with other persons, but it's not the main. 

But the special, and the self is  most interesting, the special one it's about   the physical world, if you feel it's real, I mean  feel it's real, you can be in the cartoon, but you   really feel you are in a cartoon and that is  not fake. And the self dimension, it's about   feeling my hands are my real ones, for example,  so it's about being here. Also it's about you.   So I think it's important  to have this in mind.   The technical dimension, the acceptation of  technology. If you put a headset on the   head of someone, but it's two kilos and there  are wires everywhere, it will not work. 

And it's important to measure it in  order to know if it's your experience which   is good, bad or if it's simply some technology, which is not accepted because   people don't like it. I will not enter  deep in it, but you have models to check it.   And then I will tell something which is  obvious for multi-sensory researchers   like most of you, but which is not for everyone, and  especially for cognitive psychology researchers.   It's to take into account the idea of embodied  cognition, but you are not a computer with a brain   and the brain is all, but you think, you learn,  you feel all the things with your whole body.   And I know for most of you it's obvious,  but it's not for the people, and maybe for   computer science researchers it's not  also because they see people as computers.   So this is to take into account also. So for us, I would tell if you want to build a  

sensory experiment with mixed reality, keep  in mind to take your self-presence, feeling that it's really you, your hand, but the thing  around you seems realist, at least real for you   [missing audio] accept the technology, the acceptance of technology  and about the embodied cognition to have   congruent moves when you eat, but it seems that you  are grabbing the real things, but you have your   tactile sensations which are real, because there  are a lot of experiments which can be done, and   if you train for example people to grab things in  virtual reality, but there's nothing, and after   you put them outside the box and they are not  able to open the door to grab a glass because they   have learned to do it in virtual and we don't know  more how to do it really, so it has to match.   And that's the difficult thing. Here the VR  headset can also be tricky because of access to   mouth is difficult, so many people are using  straws for example, but the straw, you know   it's cutting a part of the sense and so you  don't feel, maybe for Coca-Cola it's okay, but   for a nice wine you will lose some parts of the sensation. So I think it's important. The same, you have to know where is the food  and so it's difficult to know where are the cups   and people who will see it, or experiment  this afternoon, will see that with cups it's not yet   perfect. There are tricks to put trackers  under for, from the virtuality headset  

on the plates, on the dishes and so on, but if  you have a tracker on a fork, the tracker   is much bigger than the fork, so it's not very  convenient. And if you use a mixed reality headset  but you can see through, like Microsoft Hololens, it's good, but you cannot change the environment.  You have a very small field of view  and so you can change the appearance of the   food you are eating, but not the environment.  So it can do some things but not everything. Here you have a long list of the studies I've  found the most interesting, and what you can   remark for it - and I was not searching to have recent  studies - is that the oldest interesting I found   has three years. So it's all very recent. If we focus a bit on the topics  

they were doing experiments with blue cheese, with wine and beer, with wine, coffee, lemon cake,   juice, chocolate. So you see a bit about what  we are studying. They have changed environment,   and half was to see if it was changing  the perceptions, and the other half was to   see if we could replace sensory booths by virtual realities to have lower cost during researches.   Another thing that you can remark, is that they are most   of them with quite old headsets, which are  no more used, no more sold since years now,   so I will tell you in the next slide, but  yes, many nice things are to come because   all these studies are with, not all, sorry, but  most of them are with old headsets. And a   lot of them were using not a digital environment  but 360 degree videos, which is nice, but you   are not interacting with anything, you  are immersed in a movie, but it's not   giving all the perceptions and I think about presence, it's not so high. And there are a few modern headsets  even if the year is not so modern. So the main conclusions from this is that   the replacement for sensory booths are really  working, changing the environment of the   food seems to have significant effect,  but there's not so many studies currently.  

And this I already told you, so I think many  interesting things will come this year,   next year, and we have nice things to  see and to experiment ourselves. I will finish by what we can do. We can, I would like  to improve food enjoyment by changing environment.  If you can eat some, I don't know, some, the example is always bad, but I always have this one,   you can eat some sushi, maybe in Japan  or with an environment, a congruent environment   it can enhance this pleasure and  if it's working, why not to do it. It has been told, the idea of changing the sugar  perception to help people to have a better diet.   Can be really interesting you can do the same with the fat, with the salt, maybe we can   help the environment by making people to eat  less meat so there will be less dioxide carbon.  

So there's a lot of things that we  can do with this. And the coming things   are kind of experimental headsets, so I don't think they will come for everyone   but to experiment it's interesting. There are  virtual headsets which can give you sense,   which can warm you, send you cold air, so  this this can be quite interesting.

Yeah, thank you. Yes, thank you Jean-Christophe. And I think  now we move on into art and technology part of this.  Jean-Christophe: That's why I stayed. Anu: Yes exactly, and that is why you you stay and we invite also Bruno Mesz, from National University of Tres de Febrero, Argentina.

So please, now we sort of combine all that theory into art and technology. The floor is yours.  Thank you. So thanks for having us, and especially thanks to professor   Anu Hopia and Seppo Salminen and Kone Foundation, that made this possible.

And I will start with a little story of how we started to collaborate with Jean-Christophe   and how we designed the sort of experiments in mixed reality learning. And previously, before I met Jean-Christophe, I was investigating the phenomenon known as sonic seasoning. A little bit in the line of what Mari Sandell told before,  that is how music influences the taste of food,   in particular of wine and coffee. And we did  some studies using temporal methods because   music changes in time and wine changes  in taste in the mouth and time also   Using temporal methods from sensory analysis. And we found interesting things, the ideas, for instance  

you taste a wine and you select an attribute,  which is the dominant attribute of the wine   for instance for a red wine we have this kind  of attributes, and according to what music was   played to the people in the experiment or if  they were in silence, the proportions and the   timing of the emergence of different tastes  of the wine was really quite different So we did something similar with coffee afterwards,  that is drinking coffee with different kinds of   music, that was thought to be congruent, cross  modally congruent to bitter taste or sweet taste.   And we saw that there wasn't not much sensation  transference from, that is the sensation of the   music, didn't go too much, it didn't affect too  much the combined sensation of coffee plus music   but there was a very important emotion  transference that is the emotion of the music   was really impacting the emotion of the combined  experience of drinking a coffee listening to that music.  So when we design some devices for doing this kind of experiments   like interacting with glasses, the sound when you  drink from them.   So you synchronize the taste with the  sound or a smell organ that you can play and   launch smells and because we also investigated  smell and music experiences. So because I work  

in an art and science center so we are always  trying to combine both kinds of investigation,   research and artistic experiences. So when I met  Jean-Christophe, virtually because of the pandemic,   I was thinking of this impact of the atmospheres  on taste and in general the impact of for instance   visual atmospheres on musical experience, so we  thought of a study in virtual reality   playing the same music in different environments,  for different seasons, the summer or the winter, and   having people to write the emotions of the  music and we haven't done this yet, we want   to know if this impacts their experience of the  musical emotion or something like this, people will   can I enlarge, no, but you will see anyway, no  problem, so you play music in a summer environment. Write the musical emotions, as you see, there is some evaluations of emotions. And then we thought of combining this also with  food, so what happens if we have contrasting   atmospheres and there is some cross-modal congruence between the sounds   the visual environment and food, what  will people, how will people experience   the same food or experience a food in  this kind of environment. So this is the   experiment that we will present today, and for the, let me play, Jean-Christophe: You want to play it? Bruno: Yes. Like this.

Jean-Christophe: Do you want the first one? Bruno: The first one please. This is one kind of environment and the other one They are contrasting. Let's go further okay. And in the meantime  there was this experiment that we did with Sami Silén and Gabriel Vinderola on what happens if you play sound to kefir during the process of fermentation. 

This sound has some  effect on bacteria, so also in the   experiment today we will have environments  related to bacterial, microbial environments   like these ones, one for wine and  another for cheese, so to play here This is for cheese. Okay, so these sounds were designed by  Sami Silén and myself. We have some semantic   relations with the food, I don't want to  anticipate too much because there's an experiment   involved in this, and also the videos were made  by a Dutch artist Rob Verf, they are called 'being inside a piece of cheese' and 'the expressive life  inside my plastic body', with a 360 camera, and   they try to express the sensation of being inside,  the like assuming into the bacterial world.

And with this I give the word to Jean-Christophe. So now I will tell a bit more about the technicals, but the same as Bruno we will not tell things  which can bias the measures that we'll   do this afternoon for the presencial  participants who will do it this afternoon. So I'm on the technical dimensions. There are, we will think of this mixed reality, so we   synchronized two worlds, the real one and the  virtual one. First of all there are two parts   an artistic part and another part which is not  artistic. The artistic part is with red wine  

so here you can see real water because  Anu was not drinking wine at this time   and you have real cheese and the wine was chosen  by Delfederico who spoke just before and   we were having sort of a virtual reality part with  realistic appearance with the cube of cheese and   the glass of wine. And then when the people and  when you will pick up cheese or take a glass then   you will be immersed in the 360 degrees videos,  that Bruno has described and shown to you. So here you are in the one from the  cheese's video made by Rob Verf   and you are also immersed in a sound, in an audio  environment which has been chosen by Sami.   And also, so this is the artistic part. On the second  part of the experience you also have food it's  

a bit small in the picture, but you have here  food balls, I will not tell you about what it is,   these have been made this morning by, I  guess or yesterday, by Nanna Rintala   and so this is a reward and again we have a  digital virtual reality environment,    we have shown it a bit, I will not tell  so much, but it's looking like as if you are   in the space and you are completely  immersed, you can see planets and so on.   With also sounds, which should be congruent  with it. And we have a synthetic appearance   of the food ball. So in contrary to the, from the previous part, it's not realistic.   It looks like something else. Here it looked like  little asteroids for example. The difficulties of doing something like this is, that you have a real world, you have a digital world.

here you can see what's inside the headset for the  participant and it has too much at a millimeter position.   And this is really hard, current  headsets are doing it, but it's not so accurate  so when you will do it, maybe you will have to search a bit your food, be aware about it.   In our setup we are doing something that, for  example we are using Meta Oculus headsets and   our programming setup, but I do, yesterday an  experimental one has been released by Facebook   to experiment for developers only, so we are in  advance from them, so that's why it's    still a bit tricky and a bit fragile solution.  But it's coming, it's coming fast, but   that was the main difficulty, and to have markers  and really to make this synchronization   between the two worlds. Here you can see  better, my finger, the cheese I'm sorry, in fact the people, the remote people don't see  the marker, so here you have my finger over cheese   and here you have the same thing so you see it's matching very well but it has for you  when you press the cheese you have to feel  it at once so it's at one millimeter maximum   but it has to be accurate, and this is  not so, this is really a difficult thing For the the glass and the food   to be grabbed, so you have to  take care and the same when you pick a glass you pick it in real, but you want  that in virtually it's big, so it's difficult   to detect and sometimes it's difficult to  release. So for the people who will do it  

think when you grab your glass to open widely your  fingers, else you will have released the glass   but it will stay with you in Virtual Reality,  and you will not know really what to do.   The same for orientation in the  experiments I've described, there are a few ones   who have sensors, a bit like in the  one Bruno has shown about in the video and these   sensors can help to know the orientation of  the glass. We are not doing like this because we   don't have this kind of sensors with the Oculus  Meta headset, and so we do with the fingers but   your fingers are moving near you the less  the heads that see them, and so you have impression   the glasses will be shaking, but nothing is, no  illness, neurological illness is coming to you, it's   only the headset. So just trust your fingers,  you know how to deal with a small small glass. So yeah, I'll show this here it's perfect, but  it's not always perfect as in this photo. So this cool experience experiment  has been designed by me, Bruno and   Sami. This afternoon we will do a kind of  preliminary experiment where you will experiment  

it as you have not been biased too much by us, I  hope. We will do some measure, there will be a few   questionnaires to answer. Anu: Question time, yes thank you very much, Bruno and Jean-Christophe. These are really for the webinar audience. You have learned these experiments now by this presentation and

and those who are present here will have the  opportunity to experience those here   on site, and in real life  virtual reality in real life.   Now we have a few minutes time to discuss about thoughts and impressions on   about this session number one. I would like to hear  your opinion on the, you know, the near future   what is the near future, you said in  your presentations, Jean-Christophe, that the development of technology  is really, really fast. How do you see, what kind of, now we probably - at least what I know - the applications in this   VR, mixed reality applications, they are  a lot in entertainment. They are probably a lot   also in the business world, in which we do not now go, maybe you can   share that too. How widely these  technologies are applied now in the   different sort of platforms, whether  it is, well industrial development work or    health technologies, what  about this idea   now, when we talk about this VR dining, and you for example mentioned that might be a good tool  to help people to improve their diet, for example   or that can be used for people to sort of  enhance motivation for example for sustainable   choices and so forth, how do you see that, you know, future? How close is that kind of   or future at the moment? You can come here, yes, and you don't need to know but it would be nice if you guess. 

Anu: Because you probably have the best guess. Jean-Christophe: First of all, it's really very used now in industry,  in avionics, in medical, they  are using virtual reality, mixed reality,   but it's where it's important, where we  have money, so we have very expensive headsets.   It's also very used in entertainment with  cheaper headsets, like the one we were using   in our experiment, but the market is mature enough to make light headsets,   like a bit big glasses, that will not disturb so much. So it's really coming fast. And for example  

one month ago a very big Virtual Reality   society, HTC has made a new headset which is   for entertainment, meditation, it's very light  and so on, but unfortunately they have used all   technology and it will not be accepted because  it's too expensive for something to all.   But they are ready, next year maybe we will have a  later one, and so I think it's coming fast.   And in the same idea we are, there's a lot of how  to do sport, to meditation and I think to have nice   dinner in another environment, it's also coming  far if we develop it. So I think it will come fast. Okay, so you think like next April we will all be dining  together in virtual reality?   Not next April, but maybe it will be feasible in two,  three years. In two, three years it's really   possible because it's going fast and it  needs something acceptable. That's why I was  

speaking about acceptance of technology, which is  light, which is not disturbing you, would have no   wires because you don't want to eat with a wire...  Anu: Yeah, surely. And at the moment really   the food experience in virtual and mixed  reality world, it is very different from   from our conventional dining so we  probably need to, I don't know   we should be ready to accept new experiences.  We probably should not, this is my feeling  that we should not try to mimic the  conventional dining experience, but rather   take it as a new one, extension or  expansion of our experience.  

Jean-Christophe: I think I have forgotten to tell something very important.  Anu: Yeah, do you want to come here? Jean-Christophe: Yes, maybe I have forgotten something very important, everyone is speaking about metaverse currently, and this should be an important part of the metaverse,  if the people will join in virtual reality in   all the world's environments and if they can also  have dinner together. It will be kind of a,   sense more and wonderful experience, if you  can have a dinner with your friends, which   are at the opposite side of the world.  So we move on to session number two, which is on commensality, and actually we go back  to the world of microbes to learn about   commentary of microbes and the senses that they do   on commensality, which we don't. And we have in  our artistic artistic examples, we have them   try to learn from microbes. But before  that I'd like to invite professor Seppo Salminen

to tell us about what is quorum sensing of microbes.  Seppo Salminen from University of Turku, please.  Thank you Anu. Thank you for the introduction and now we go from   dining of humans, to dining of microbes. And that's actually telling you something about   the two and a half kilos at most, one and a half  to two and a half kilos of microbes every one   of us is carrying in our gut. And those are the  microbes that are discussing with us, and we are   discussing, we are responding. And if they are the  right microbes we don't have to think about them,  

if they are the wrong microbes, we feel quite  quickly that something is wrong.  So, quorum sensing in the way is the process of  cell to cell and cell to human communication   in bacteria. As said, bacteria will talk with  each other, that's the quorum sensing part.   But they will also talk to us, through our immune  system, through our gut, through epithelial surfaces,   and we respond. And it's quite common  to forget that we also respond to those questions.  So for a long time we believed that  the bacteria are just quiet, silent and not   doing anything. But in actual fact the more we have,  the more refined methodologies we have, the more   we understand that there is intercommunication,  high level of intercommunication between the two.  

They produce signaling molecules and  these actually help us to understand the   communication, but they also you define how  the bacteria act. And this system is called   quorum sensing. So, regulation allows bacteria  to have the so-called social life    based on the control of the component secrete.  So, all of this is actually taking place by   signaling molecules. And what we are trying to  do we are trying to understand the change in   concentration of signaling molecules and how  they interact with our genetic transcripts,   and the transcription of other microbes. This  interaction is actually seen in many gut bacteria.   It's also seen in bacteria  that we use in foods, and if we think about quorum sensing  in bacteria, low cell density is   something that we should perhaps avoid. And high cell density is something  

which makes the group behave in a certain manner.  And we can modulate that manner by eating.   So, dining with each of us, we are actually dining with  our bacteria at the same time and the bacteria are   dining together. So it's a real community  working on dining and nutrients that we get.   Then fermentation. One of the important things,  that we forget today is that fermentation has been   most important for the survival of mankind. If we go  back in time, about 60 percent of our food supply   has been fermented. So we have had a huge amount of  bacteria ingested every day. And now we are doing   our best to actually make our food sterile.  We pasteurize, we UHT-treat, we even sterilize food.  

And we think about for instance the first food  we have, with this mother's milk. Mother's milk is   full of bacteria, viable bacteria, dead bacterial  cells, bacterial metabolites, signaling molecules   for the microbial community. And  if we compare it to infant formula,   practically no bacteria, no metabolites, no  bacterial cells and it tells you already   that that kind of introduction to our food  system is going to have a profound effect.   If we lose the fermented foods like  we have lost them in the western diet   it's also important that  we lose the bacterial   input, and so the quorum sensing part is going  to be quite different as well. Fermentation gives us dynamic stresses, and lactic acid bacteria were taken out just as examples by Lucrecia,   and when we do fermentation, there are several  different steps that actually influence the   bacterial signal. Fermentation makes the bacteria produce not only aroma  

compounds, but these signaling molecules, that  influence the transcription that we see in the genes   and the way they behave. Of course at the same time you can produce   different flavor profiles. You can produce  different industrial properties, but the main   thing is that fermentation is something that we  perhaps need more and more in the future again.   This has been noticed by the European Commission, and they are now actually   funding a lot of research to fermentation and  especially plant product based fermentations.   It is in the European Union goals for 2030  to have a significant amount of plant-based   fermented foods in our food supply. So, what does  it mean? It means that the bacteria actually have  

a significant role in fermentation control. We  have seen it in our experiments ourselves.   Bruno was mentioning the kefir. And the South-American  sweet kefir that Gabriel Vinderola has been   studying with us, and later on short-term  fermentation with yogurts. And this is  

something where we need to continue, and  also understand that if we have simple   things like sound or music, how they influence the fermentation control.   And perhaps they are also important in producing  specific sensory profiles and at the same time   they could be an ease for the industrial process.  We have seen this for instance in beer production,   if we give specific sounds to beer, when it's maturing we can   provide different flavor profiles  and different sensory experiments   have proven, that the modification of  beer flavor is actually quite possible   just by the simple use of different  sounds or different types of music.

So, what do we know about it now? If we look at  early human diet, there were high numbers of   microbes in the diet, and they produce the  high level of interaction with the immune system.   Priming the immune system towards health. So  what we have done? We have industrialized the   human diet. Very little, very few microbes  and no levels of interaction with immune system.   So we are priming us for the situation where the  simple pathogen can cause a lot of harm because   there's no resistance in the normal microbiota. So  what do we do? Microbes supplemented diet as  

even the European Commission is suggesting.  More fermented foods, new types of fermentations,  and going into actually  plant-based fermentations.   Just recently published in Cell, one of the top  journals in this area, indicating that if we do   replace even high fiber diets with fermented  food diets, we are actually improving health and   well-being quite significantly. So this all in the  face of bacteria discussing with each other and   discussing with us. So perhaps that also gives us  some future directions for nice research. Thank you. Thank you, Seppo. This quorum sensing  of bacteria, at least for me, has been really  

one fascinating new finding. I have  understood, I have learned new things   during this project. And now I would like to   move onwards in this session number two, into art and technology where our researchers and artists together have been inspired by, for example this  quorum sensing microbes, and built two different types of VR, I don't know, how would you call them?  Are they VR, are they mixed reality? How would you call this quorum sensing and tweeting the feelings    examples that I hope we will hear now presented by Bruno Mesz and then Sami Silén.

Okay, thank you, Anu. Another classification  of this experience, I really   asked Jean-Christophe, he's the expert,  it's not virtual reality for sure, so this experience that we are going  to present today after the webinar, is   is inspired by this, also these experiments  that we have been doing with Gabriel and Seppo and Sami, on fermentation of bacteria, the effect of sound of fermentation, so we were inspired by this special bacterial sense by which they detect the presence of  others, and they change their behavior,   turn in to a collective way of behaving And to the metaphor of this in   the situation of commensality, in the situation  where we have dinner together this part, yes so the idea is that to metaphorize this special sense at the human level by invoking the sense of  touch, of human touch, because we reach and contact   the others by this kind of, by this sense,  and also in this installation we try to   increase the sense of remote presence of the  other people that are dining with you   so how do we conceive this? Well, there are many layers involved in this installation. So first let me skip this and go to the  scheme of the installation and then I will   go back to this. So the idea  is that you have a special soup bowl.  

And the bowl has an embedded tablet in it. And I'm going to return to this. And you eat the soup using  a spoon, that has an accelerometer.   You see the basis of the spoon, the second picture,  you have this capsule and there there is an   accelerometer that measures the force, that you  apply to this console. You eat and this force is   detected, and the signal of the accelerometer is  controlling sound, is controlling the factors in   a computer, that regulate sound intensity.  But you don't perceive it as a sound,   but as a tactile stimuli, because we send the  sound to these special belts that have transducers   that vibrate in response to the sound. So when  when some somebody eats at the table with this   spoon, the other people feeling their backs the  result of the activity of eating, of this person.  

And it's more intense as it depends on  the intensity of the activity of eating of this   person, so you can eat in darkness but nevertheless  feel the other people eating with you via this   extended tactile sensitivity metaphor of quorum  sensing the others dining with you.   But I want to return to the soup bowl  because there is another layer of meaning or another idea in this that this using  tablets or using digital devices at the table also like contemporary art or art in general  maybe have different layers of meaning that   are not strictly related or strictly consistent,  but they're like different concepts that   mix in some way in a piece, in an  installation. So, the the soup bowl is based in   an experiment of on cross model correspondences  and I want to show you a little video on this.   Video: we present conceptual design of tableware  that explores the use of digital tablets   embedded in greyware for the creation of  multi-sensory gastronomic atmospheres.  

And it was conceived from the results of a pretest  on crossmodal associations of shapes, colors   smells and materials with four emotions  induced by music. Energetic, tender   in awe, "sobrecogido" in Spanish, as in  the experience of listening to pipe organ   music in a temple, for example, and  the emotion of being agitated. Each video is accompanied by a good  smells presented in on perfumery strips.   Gas and dollar bill smell for energy, pear and apple for tenderness, absolute roles for awe, and sea smell for agitation. 

We are currently analyzing the results of a new test and plan to use them for   the design of real plate wear or for designing  the environment at dinners in virtual reality.   So that was the experiment so people  listen to different musics and they draw   shapes inspired by the music, they selected  materials from at list, for instance is   this music related to wood, to  metal, to stone, to water, to liquid,   and also colors. And we are now, this was  an installation in a museum, and we are now   here, we are going to present the actual plateware  inspired by a musical agitation. So this is  

combined with the quorum sensing. We present like two ideas, more than two in the in the same installation  Another thing is that we want to reflect on the use of digital   devices at the table which is generally regarded  as negative because we are concentrated in the   tablet and not interacting with other people, so  they are like interrupting social interaction   and commensality, but here we embed the tablets  in the plateware. They are used to generate   a multi-sensory atmosphere with sounds and  colors that are shared by everyone in the table. This is some photos, Sami has nicer  photos I think, and he is going to talk a   little bit about the technical implementation  of this quorum sensing installation.   Thank you. Okay, I'm Sami Silén, and I'm a PhD  student here in the University of Turku.  

Bruno already told some technical things about  our quorum sensing system, but I'm gonna give   some glimpse and maybe tell more about the  process how this thing has been evolving. The system is based on basically three  objects which are the spoon with the motion sensor   attached, and then we have a vibrotactile  transducer or actually two of those inside   the cloth belt that you can see on the  on the left, and then we have this glass bowl   and a plate on top of that and we  call it, like Bruno told, agitated plate. In the center of this system is a motion sensor  that is attached to the spoon and it holds   an accelerometer, a microcontroller  and a battery. And this thing tracks   the movement of our hand and converts  it to a sound and vibration. But the system is not that simple.   We need something else also. And it's based,  like Bruno told, on a code on a computer, but  

we need several kinds of things and programs. In this system we are using   four different programs, or should  I say programming tools which are called   Mosquito, Arduino, Processing and Ableton Live.  And all of these required some programming and   making some codes how these motion  sensors will react with the system.    And then of course we are in a wireless world with  the motion sensors, so we need a wi-fi router for   having this network because the devices are of  course talking to each other wirelessly. And then   from the computer we go out with the audio device  that leads the sound to amplifiers which    supports the sound and amplifies the signal so that  it can be used by tactile transducers, which looks   like that. It's a kind of special speaker which doesn't have this basic system that   the usual speak

2022-01-24

Show video