4 - Graph Databases for Embodied Conversational Agents in Unreal Engine 4
okay thank you for the very nice introduction and yes my talk would be about using the unreal engine 4 and neo4j in combination so that we can build awesome applications with embodied conversational agents so what is fantasia fantasia is what we were talking about today and it is a plug-in for the unreal engine 4 that allows connecting neo4j among other things to the unreal engine unreal engine in order to build embodied conversational agents so let's build the first of all a bit of context around fantasia and let's talk about meta-humans so many humans uh i could be one of many an exciting new technology uh brought forth by epic games and they are photorealistic humans you can see in the video playing here they're visually stunning and it makes sense to back visually stunning virtual humans with believable interactive capabilities that must be supported by ais and in this case we will be talking about graph supported ais to make these meta humans more believable we will be seeing a couple of examples with other types of virtual humans but what will what we will be seeing will be transferable onto meta-humans very easily so the the topic here is how to integrate neo4j in the unreal engine so that meta-humans are just the front-end towards what happens at the ai level and in general integrating neo4j in the unreal engine opens several interesting applications in different fields for example you could think about using the neo4j connector to build a visualization visualization metaphors for graph data using a real-time interactive 3d technologies provided by epic games so fantasia is the an acronym of framework for advanced natural tools and applications with social interactive agents it is a plugin for the real engine for it is composed of a single single components that can be used for other types of applications so the neo4j connector can be used for anything it is research oriented it is thought to support research in human computer interaction it is totally open source you can download it from github and i will give you the link in a few slides and it allows to integrate neo4j in the unreal engine 4. so how does this integration happen the unreal engine 4 for those of you who are not familiar with with it provides a visual programming language called blueprints and you can see that in this editor we can see the event graph so it is true graphs are everywhere the goal here with fantasia is to integrate the capability of accessing neo4j inside blueprints so that this is very easy to design applications that take advantage of the technology from neo4j specifically fantasia adds two no two main nodes to blueprints and a connector called neo4j that allows you to submit queries to neo4j and get results as soon as these are ready through these nodes so you submit queries so you specify a neo4j component you bring a cipher query inside this field string field you can perform different kinds of operations the most common in single request but you can also open transactions add queries to transactions roll back transactions and of course commit transactions if you're working with transactions you should provide the id of the transaction that is also returned by neo4j and the the submit queries function also supports parameters so you can provide maps of strings onto other strings as input to this function and you can use parameterized queries here in the query field to make them run as soon as the results are ready an event is fired because the unreal engine is a game engine has an event-based architecture and you cannot connect application logic by using the results that are coming from here so mainly you get rows of the table that neo4j returns as a result of a query so how is the fantasia architecture organized of course there is the unreal engine here managing input from various sources and devices of speech webcams etc though it is also integrated with through fantasia with third-party services and we will be seeing something from microsoft azure and amazon ws internally the logic the application logic processes inputs and uses a bayesian network this is the next component that is going to be published in the uh in the repository it's still in development and it uses the internal systems of the unreal engine to control virtual characters in our uh in the way that we use to uh apply fantasia to deploy fantasia in different domains we usually take data from different sources so for example wikidata or flickr or the internet movie database and framelit and we build graphs in neo4j that are then extended either by external uh algorithms or even using the the great gds library so after collecting the data we enrich the data and at this point the unreal engine can provide information to neo4j about what the user is doing here and he can also query you know for j in order to understand what it should be doing as a response to the input provided by the user so neo4j informs the unreal engine about how the interactive application should proceed before going actually inside the unreal engine and see the inner workings of a couple of demos that i have i will give you a couple of links so you can find most references here and the wordpress page you can find links to the social medias uh you may be interested especially in the youtube channel where there is a series of tutorials that will cover the first example that i'll be showing you today the second one is a new never has been shown around before and you can download the plugin from here from this github link and as i said it's open source it's free to use you just need to have fun with it so let's talk about the example applications that we want to be uh using for our demo the first one is based on the movie database included in the tutorial from neo4j that we all uh love because it's probably the first thing that we did with neo4j and this is a genesis 8 character called jason who looks a little bit like dr manhattan and cortana's son and i will be showing you what is the final result that we want to give and then we will dwell inside the inner workings of json so who acted in the matrix there is a bit of latency due to speech synthesis and animations people who acted in the matrix include emily from hugo weaving lawrence fishburne carrie anne moss keanu reeves tell me the movies keanu movies keanu reeves appeared in include the matrix the matrix reloaded the matrix revolutions the devil's advocate the replacements johnny mnemonic something's gone together okay so in this first example we will see how we can build uh an unreal engine 4 application that accesses neo4j to recover information about the movie database and this is the one that is also presented in the tutorial series on youtube the new one that i want to show you is this one you is made using the default mannequin in the uh in the unreal engine as we call bastion in this case and this is an application for conflict detection in sequences of commands so we are using neo4j to track sequences of commands in the cooking domain in this case and to detect if there are conflicting commands in the sequence and we use information from neo4j to build the appropriate error messages that are thought to uh better explain the error that is occurring so that the user can solve it the video is in italian but there are english subtitles here that is so using neo4j in this case the uh the agent was able to understand that the the butter cannot be cut because it is a liquid and it has become a liquid after being exposed to heat so uh that is the offending action the conflicting action that is signaled through a negative polar question a negative yes no question bastian here in the application is part of the phd thesis of my friend and colleague dr maria dimaro who is a computational linguist and the title of a phd thesis is shouldn't i use a polar question proper question forms disentangling inconsistencies in dialogue systems if you want more information about this you can read this recent paper that we're publishing or you can contact her with the expert in the in the field so before moving to the actual questions let's move inside the unreal engine and first of all i would like to show you how a real meta human looks like so if we can understand what we're talking about i am a metahuman the next generation of digital human powered by unreal engine metahumans are high fidelity digital characters created by you the user on our new content creation platform metahuman creator okay as you can see it definitely makes sense to build advanced ais for this kind of artifacts so let's start with json and let's dwell inside the json blueprint that we have here the logic behind json works like this so this is a blueprint and this is the event graph the event graph describes application logic linked to events in this case we are listening for an event that fires when i press the state the space bar and it starts a natural language understanding in this case using microsoft azure as soon as i stopped talking the azure nlu component which is included in fantasia fires this event with all the information the information that is needed to control the feedback from the from json and here you can see the neo4j component that is used to configure the connection in this case it's a local connection with the simple database and what we're doing here is using the top intent so the top intention recognized from the query to access a map of queries that match every intent from each user onto a cipher query in this case i'm assuming that only one entity will be pronounced by the speakers is for the sake of simplicity so i simply access the first element of this array of entities returned from asia and lu and i compile a map of key value pairs param and value where the value is the entity recognized and i pass it through the parameters input here the param input is present in both queries so in one case i want the list of people who acted in a certain movie in the second one i want a list of movies in which a certain actor was in both have a single parameter and both return their results as element how do we organize this on azure this is pretty simple and here's the interface from microsoft azure i define two types of entities list entities and i put all the actors and the movies that are included in the neo4j database in here it is pretty simple since data can be uploaded using the uh json format to convert results of coming from neo4j into this format so that you can upload knowledge that is included in neo4j directly inside microsoft lewis here and the important thing here is that when you define the intent lewis is automatically able to identify substrings that match elements inside the list so in this case whenever i send audio to microsoft lewis it gets the transcription it is able to automatically identify entities and return them back to the unreal engine so that we can build the cipher query on the basis of the results of microsoft azure in this case we're using a single request and as soon as neo4j returns this event is fired the first part of the answer is assembled so depending on the intent people who acted in movie title include or movies actor name appeared in include are appended to a comma separated list of values that are obtained by cycling through this array here of results of rows sorry each row contains a map of strings on two results so these are the cells we know that in both cases the element is the cell it is the column name that we want to to obtain we convert it to a result cell simple also nodes and relationships are supported as result cells in this case we only need the strains so we build a comma separated comma separated strings uh containing the actual results we could have done this directly in cipher but just to show you how to iterate through the results of the neo4j this is okay and as soon as this fourth loop finishes the result is sent to the tts and we are using amazon poly here together with lip sync on and we're using amazon poly because it is the only one among the available ones that also provides visines that that are then used to uh animate lip sync using this plug-in texture lip sync so this is covered in the uh tutorial series as i said so i would like to uh use this just to show you uh briefly how an application based on neo4j looks like inside the unreal engine and i would then now like to move on the other example which is bastian here which is more interesting and shows more advanced possibilities that you can use with fantasia and neo4j that have not yet included in the tutorial series so the basic mechanism is the same we have asia nlu that controls natural language understanding of course in this case the processing of the answer from asia is a bit more complex we will not go into the details of here suffice to say that information coming from azure nlu is used to build a frame so frames are a concept that is used in dialog systems that describes actions and slots so objects that must be put in objects must be put in the slots so that frames actions described by frames can actually be executed once we have an internal data structure that represents the frame we can add this action to the common ground which is what the agent will be reasoning upon and this is where neo4j comes in so using the frame we build a query to create a new action that must be added to the sequence of actions that is part of the common ground inside the application and in this case we don't want to accept the action until we are sure that the action can be performed performed depending on a set of preconditions that must be checked before accepting the action in this situation we use the transactions mechanism to simulate the hypothesizing mechanism of the agent what would happen if i accepted this action so what we do is we create the query we create the parameters and we begin a transaction we open a transaction in neo4j when we open a transaction the output event here will give us the transaction id that we can use to run other queries inside neo4j so before under going to check what checking preconditions means let's move inside neo4j and let's have a look at the data that are in the graph while we have unreal interact with neo4j as soon as the dialog starts we have a user everything starts with a user a user declares actions so sequences of commands can be represented as chains of commands linked by is followed by relationships an action is a frame instance and it also refers to and creates other things we'll get back to this let's concentrate on frame instances a frame instance is an instance of a frame that is imported from frame net frame net is a linguistic resource describing actions and then slots and it is available inside neo4j here so we have the entire frame net inside neo4j but we're using this sub-graph designed for application logic to uh describe what we're actually using of frame net inside our application specifically we are using a patient and an instrument so what is being ground because this is an instance of the frame grinding and with what we are grinding it although we are not using in using them in our example there are constraint nodes that limit the type of objects of entities that can be put in the patient slot here so this constraint refers to an ingredient and an ingredient is linked to the root of a wikidata tree that describes edibles so all the objects in wikidata that descend from the edible category are ingredients in this graph and using the intermediate graph of the application logic we are linking together frame net and wikidata while not altering the original structure this is wikidata this is framelit and we use intermediate graph to link them together so here we have pre and post conditions that help the application run precondition nodes contain cipher queries as properties yeah and in this case these are parameterized queries in this case this node describes a precondition that states that a certain element in a certain slot should not have specific labels for example for the grinding frame we know that we cannot grind liquids and we cannot grind the powders so in this case we have this query and we have two relationships that provide the parameters that should be used for the query so patients should not have a liquid and the patient should not be a powder together with an explanation that is used to generate natural language for natural language generation all the preconditions share the same output structure so here we have an eval field which is a boolean field describing if the precondition is verified or not we have a conflicting action if it exists is there an uh an action that is responsible for causing inconsistency the nl explanation is obtained by the property that we just saw on the relationship the conflicting frame is useful to retrieve the verb that is used to build the error message and the original entity is the violating entity in this case this action through lewis we know that it referred to an entity called the muscata or nutmeg that was assigned to the patient's slot this object refers to a perceived entity so there is a difference between between what we perceived and what we name that was the nutmeg and since the this action was accept acceptable the post condition of grinding creates a new entity that has the same name from the preceding one but it acquires the powder label so if i were to say once again grind the nutmeg this would be the only perceived entity available because of quantities are considered too but this has the powder label so it cannot be used and since there is it is also possible to understand that the powder label was acquired by applying this action to this object that did not add the powder label one we can what we can say is that this action should cannot have been performed if you want to grind the nutmeg again so this is the knowledge graph at behind the bastion application let's go back to unreal and let's see how we can check the preconditions so the interesting part here is that from the unreal engine point of view a single query is sufficient to check all the preconditions because this query is designed so that it takes the last action that is introduced inside that is represented inside this transaction takes all the preconditions of that action and then it used it uses apoc to run all these queries and since we know that all the precondition queries share the same output structure we can obtain at a single table that checks all the preconditions and simply provides back to the unreal engine the situation as it has been computed so let's go back here to the event graph okay thank you and in the last part what we want to do is to consolidate the common ground so after the preconditions are run inside the unreal engine the results are passed through this function that simply verifies the preconditions and what happens here is that if all the preconditions are verified then we can commit the transaction by applying the post conditions so as you can see here i'm specifying the transaction id again and specify and commit transaction and i'm using this cipher query that resembles a lot the preconditions one that simply collects all the post conditions and runs them but if this condition is not verified then i roll back the transaction so nothing is materialized in the database and i use the information that is coming back from neo4j to build the appropriate polar question then some information about prosodic realizations are introduced here and everything is sent back to the tts again so this is a more advanced use of neo4j inside the unreal engine it uses again the azure nlu and hrtts in this case in the other case we had to use the amazon connector because we needed lip sync for the uh json application and in general this is what you can do this is the potential that neo4j has when linked to a powerful technology for front-end realization with real-time interactive 3d the last thing that i would like to do now in the remaining time is to share with you the promo video of fantasia so that you can have a high level look at what you can expect and i hope you enjoy fantasia that you download it and that you have found with it there are so many things [Music] so many things things i would like to tell you things i would like to show you things i would like to know [Music] things i would like to be do you really want to see me don't stop at the looks come beyond [Music] [Music] [Music] um [Music] do [Music] honey [Music] [Music] um [Music] [Music] so you
2021-08-08 08:45
