2023 06 21 Aries Working Group
Sam Curren (TelegramSam): Awesome welcome everyone to the Sam Curren (TelegramSam): twenty-first of June 2023 areas were a group call. Sam Curren (TelegramSam): Got some cool stuff to talk about today. Pretty excited about it. This is a hyper ledger call, and so the antitrust policy in the code of conduct are in effect, here in the notes there are links. Please reach out to Steven or I, or any hyper ledger leadership. If you have issues and we will get them sorted so they can not get in the way of our work. Sam Curren (TelegramSam): the agenda is in the chat, the gender link. You're welcome to make any changes useful to the community. Is there anyone new today that would like to introduce themselves? Sam Curren (TelegramSam): We are glad you're all here. Sam Curren (TelegramSam): announcements. It looks like there's a new announcement here for the areas marketing working group. Helen Garneau: Yeah. Hi, good morning. Everyone. My name is Helen Garnell. I do. Marketing for Indo. I also chair the
Helen Garneau: Marketing Committee for Hyper Ledger with the membership here at Hyper Ledger, and I've been working with Alex Helen Garneau: Metcalf and some other folks in the community talking a little bit about kind of the outcome of the last few Aries calls where it sounds like there is a need to update Helen Garneau: messaging, branding descriptions the whole, the whole kit and caboodle when it comes to dairy's brand. So we are, gonna get together in a, we're kind of a a working group. Maybe it's a task force. I don't know whatever the appropriate Helen Garneau: languages for for this type of or a group. But get together and start working through some of the updates. regarding branding. So if you have an interest in contributing to the discussions. Interest in adding, you know, chain anything from everything's on the table, anything from descriptions, Logos, whatever you think would be helpful. please attend. If there's somebody else on your teams from your organization. That is is a marketer who might be interested in joining. We, you know, open arms for anybody who wants to roll up their sleeves and help out with this these items. So I put the information there for the group that we're going to meet next Tuesday, 9 and Pacific. Helen Garneau: And there's the zoom link. I am on discord. So if you have any questions or thoughts or whatever ahead of time. Please don't hesitate to tag me there. In the areas groups or direct message. Alex, did you have any other thoughts or call outs at this time for that. The announcement of the group. Alex Metcalf: not for the group. It's an activities going behind the scenes, Bob, probably save that for the working group meeting. So yeah, please come in. And interesting, contributing to any level, it could be very high level as well communicate out the chain. Not just def specific stuff. So all input is welcome on Alex Metcalf: talking about and sharing the great work that is being done here Sam Curren (TelegramSam): also, lurking is welcome. If you're just interested in listening. we have a bunch of folks in the community that are speak up a lot, and we have a bunch that just listen. And every type of interaction that you desire is is perfectly appropriate. So
Sam Curren (TelegramSam): thank you. Sam Curren (TelegramSam): Helen and Alex and anyone else involved for your work, putting that together and your work on the marketing stuff. Helen Garneau: for sure, for sure. Helen Garneau: Oh, I saw John came off mute. John, did you have any kind thoughts on this.
John Jordan: No, I just John Jordan: my no, my wife puts it on mute because I can't unmute myself. Oh, no worries. So I just sit. I alert quietly, with mute off, hey, we we love Lurker, as we said, Okay, thank you. Bye. Sam Curren (TelegramSam): awesome. Very cool Any projects. Want to highlight any updates Sam Curren (TelegramSam): to their work. Timo Glastra: I don't know if I Timo Glastra: already shared it here, but it isn't updated in the Timo Glastra: in the notes. I see Stefan is, is doing the work. We have released Arizona Javascript here for 0, I think, 2 weeks ago, with a lot of new features have now completely removed the need for dependency on in the SDK. everything is plugable support lights, agnostic anocrats, and also opening for pci issuance. so yeah, big release Timo Glastra: Nice.
Sam Curren (TelegramSam): awesome. excited to hear that and good work. Thank you for your efforts. Sam Curren (TelegramSam): Other updates. Stephen Curran: All right, we'll have some some cool stuff along mute button. 0 8 2 acupy is ready to go, probably. I'll do an Rc. One today and get that out. So we did. We didn't. Our C. 0. had a couple of things we wanted to get in it, so there'll be another release of a could buy in the new future. Sam Curren (TelegramSam): Very cool. Sam Curren (TelegramSam): Thank you. Steven.
Sam Curren (TelegramSam): Awesome. Okay. Here's topics for today. Alex, we may not need to do the areas marketing update unless you want it to additionally given the presence of the call. But I wanted to at least call attention. There. Is there anything you want to say in this meeting specifically about that, or Alex Metcalf: I'll give you like a 1 min update for sure, and then we can carry on with the majority of that material through to the working group call. Sam Curren (TelegramSam): You've got a minute. Alex Metcalf: sweet. okay. to say that the just to update the deal that things are moving forward. really? Well, I spoke to several of you in the week. Alex Metcalf: You could sell this and Helen and Sebastus, and looking at Alex Metcalf: the the link. There is going to be a working documents which is gonna work so bottom up. And some of the materials we need. But just suffice to say, that
Alex Metcalf: is, bring up some really interesting questions as to to do this exercise, and to think how we present ourselves, and how we have a better Wiki landing page, and how we by the page and the hyper light, and all those kind of things. Alex Metcalf: It's being questions as to what Harry's need to be presenting itself as as it's a Samuel. They what are the 3 things it does great, for example, that would be amazing highlights of people coming in that would be differentiators. So give me some exercises around that. Helen's got a great question that we'll probably use come Alex Metcalf: next week. Alex Metcalf: But if I say, you're welcome to look at that document for updates. I'd be putting some more things in that'd be working off of the need to go back into there. But if you go, any interest in how you present ourselves, if you or any parts of like
Alex Metcalf: of things that we could say better or materials that we need or videos, we should highlight or any aspects of this you can reach out to me, or you can use this document or come along even better to the first being that I just mentioned. Alex Metcalf: So Moving along with a focus on what's the low hanging fruit to get some more accurate information up first, and then we might refine that later. So let's get some things up to say what areas is doing now, because lot of information out there is is 2,019 era from, you know, announcements of what this is, and things have moved along a lot since then. Alex Metcalf: And now is that time to shine? So yeah, thank you for all the people I reached out to you and hope to see no of you next week. Sam Curren (TelegramSam): yeah, thank you. Alex. I will credit Alex for making me think about things like, how to properly articulate. what is different about Aries than than other agent projects that did used to be a question, because there kind of weren't any other agent projects. And so that is that's something that's made me think a lot. So so they're they're doing fantastic work. Sam Curren (TelegramSam): okay, quick, update there the stuff we have on the I guess next next week is not mediators, because this week is mediators. I bet. Edit we have a a couple of main topics. Sam Curren (TelegramSam): Oh, one that's not in the list, but but is another quick update is the open wallet foundation update and then we want to talk about mediators. and the the the did peer, unqualified. Did mediation update. There's now a migration, Doc. which would be awesome.
Sam Curren (TelegramSam): so so yes, that is the that's the goal there. any any adjustments we want to make Sam Curren (TelegramSam): before yeah, any adjustments. We want to make it to the agenda before we get going. Sam Curren (TelegramSam): okay, owf, update. last week we you know, I shared a a link for a sort of a a proposed collaboration type of a statement. and we're still working through. I say, we Alex has volunteered to to be involved in that more directly. and so we are, still, having conversations with auto be of leadership to figure out the Sam Curren (TelegramSam): sort of the right thing and path forward there. And so it's happening, although a little slower. there's a little bit of confusion in there, and we're working to resolve that. so yes, and Timo any other Sam Curren (TelegramSam): comments. Another. Vf. Timo Glastra: yeah, maybe 2. One question on like you said, like, still in discussion, I think
Timo Glastra: owf like in, like their Timo Glastra: already partners, with high pledge and a and so I think it was kind of finished right? The discussion at least that's what I heard from. Oh, owf! site! Timo Glastra: But that's not true. Sam Curren (TelegramSam): No. Well, it's it's let me try and explain a little bit. There was a a little bit of a misunderstanding about the purpose of the. It's sort of a joint statement that we are proposing. There isn't any legal changes actually necessary between the organizations. There wasn't any actually in the beginning, anyway, because of the compatible licenses. being used by the different organizations. Sam Curren (TelegramSam): the point of the statement was more of a signal of like Sam Curren (TelegramSam): collaboration, not necessarily the establishment of anything like legal, necessarily. And so there's a little bit of confusion early on about what the intent was, or whether something was actually needed. And and that's that confusion is is being resolved and and unraveled a little bit as we go.
Sam Curren (TelegramSam): So the the their response was, based on a legal perspective when they're not wrong. but the but our goal wasn't really to us. That was something legal, either. And so it it also wasn't really completely the thing that we were trying to do Sam Curren (TelegramSam): and and so trying to resolve the the confusion there. And I think we're based on emails yesterday. that, I think we have. Even up to yesterday evening we have some increased understanding. There. Sam Curren (TelegramSam): does that help T-mo Timo Glastra: or any other comments there? Timo Glastra: I Timo Glastra: not 100% happy yet, I think with outcome. and
Timo Glastra: that's okay. I think because we don't have to all agree on on like the the outcome. But I do think it's still interesting to pursue, and I think I would also. And this is something we would have to discuss, for example, in Timo Glastra: for example, air soon with Javascript community. But if there is like if Timo Glastra: the open Wallace foundation as a whole isn't ready yet to host a whole project like ours. Like we could try moving single projects over, for example, and see how that works. And if we can make that work and do that one by one and so like, I can't speak for the whole Afgh community here, of course, but Something that I would be interested to, and, for example, for seeing you like being a guinea pig project that Timo Glastra: that we can try things out with, I think. because I think the main thing we have set here, like owf is not ready to Timo Glastra: like. It's very not mature. I think I heard from owf. Also there they've offered to to basically hire everyone on the hyper ledger Timo Glastra: like that can help in how I that helps with hyperbolic on the infrastructure and operational site that they have offered to hire them. So I think that that should give
Timo Glastra: quite a lot of assurance. but yeah, just wanted to try it out here. so I one thing that I want to highlight there is that I I wouldn't really consider Sam Curren (TelegramSam): what we've proposed is sort of a final outcome in any way, but rather an intermediate step that can be something that we do and get done. but that doesn't necessarily pre preclude any future actions there either. This be the the the nature of these things is as as I I've heard Steven say a lot is is a doocracy in the sense that that the the people who do have the most control over over what's going on. And so the Maintainer of of any individual project or Sam Curren (TelegramSam): or library matter the most when it comes to sort of what actually happens. You know individually there. we are a voluntary coalition of of of of projects that are together, etc. And and I think that there's some there. There's some helpful pieces there definitely, some discussion to be had, and I and I think that any I think the progress that does move forward would be substantially easier if it's a little bit more gradual. and that allows for Sam Curren (TelegramSam): the the change to be metabolized in a in a in a, in a easier way. There are some open questions there. And so definitely, there's ongoing things. I I don't really consider this to be like a done and Sam Curren (TelegramSam): and Sam Curren (TelegramSam): and in no more discussion but rather the change. The nature of the discussion. in a way that allows us to
Sam Curren (TelegramSam): to accomplish things in a more targeted way as we desire. So what you described there, Timo was not really at least in in my brain, out of scope for what? What's possible in the future? Sam Curren (TelegramSam): Any any other comments or questions from anyone on that particular topic. John Jordan: I will be having a try to have a discussion with Daniel tomorrow. If John Jordan: there's points people need to John Jordan: cover or not cover. I'm happy to receive those Sam Curren (TelegramSam): cool.
Sam Curren (TelegramSam): Yes, please reach out. Thanks, John, for for for being involved, and and and you can reach out to John for Sam Curren (TelegramSam): anything that you would like to share. Sam Curren (TelegramSam): John, you're on discord, I do believe. John Jordan: Yep. Sam Curren (TelegramSam): And so John is reachable. There. Sam Curren (TelegramSam): cool any other questions, comments.
Sam Curren (TelegramSam): okay. Sam Curren (TelegramSam): next, up on the agenda here today is mediators. and I wanted Steven. I brought up the concept of Sam Curren (TelegramSam): a Sam Curren (TelegramSam): of having a sort of a mediator status update round up thing. And so I'm gonna check this over to him for an intro there. Stephen Curran: all right. I'll just do
Stephen Curran: an intro to the problem. is my thought. Just so. We have a a foundation for for discussion, and we'll go from there and see how things. hopefully, this helps. Stephen Curran: I all I am is a messenger of the problems. I don't have great solutions, so Not going to help there, but I'll show you where we've got to and our and are talking about it. Stephen Curran: everyone knows what a mediator is. Mediators are used for wallets. In particular, they can be used elsewhere. Actually, we're we're. Stephen Curran: I had a brief talk this week about whether we can use mediators to get rid of the developer problem within Brock. Because. we're finding some of our developers work for companies that don't allow the use of end rock. Stephen Curran: And we could actually use mediators for so me, if not just for Stephen Curran: Wallace, but that's the biggest use. Stephen Curran: for scalability. We want to be able to use a container orchestration platform, some horizontally scalable platform for using it. Scale. the number of instance of a mediator. As the low growth increases or decreases.
Stephen Curran: support migration of instances. So from time to time a platform scheduler, like open shift, will suddenly say, Hey, I'm gonna shut down this node. So everything is on. It needs to get migrated somewhere else, so instances would be shut down and started up elsewhere. That's gotta be supported. Stephen Curran: we need queued messages? Stephen Curran: so that when a instance is shut down or started up, or when wallets are not connected, to messages are not lost. So they're not sitting in a in a memory queue associated with the instance, and then get lost when that instance disappears, for whatever reason. Stephen Curran: And then the other thing is, we want to understand what scalable means for any particular solution. How many wallets can we actually support? what are the assumptions about? How many active wallets there's likely to be for a given use case. We've had lots of discussions around the Vc. Wall, and on that case well, if we add, you know Stephen Curran: a a population of 50,000 users. Well, how many are going to be active in any one time? How many? How many messages are we going to be able to process for those? What does it need for there to be 50,000 wallets out in the wild? Stephen Curran: in terms of how scalable the Stephen Curran: mediator is. So I've got pictures to follow and Ares example. And then mediator examples and the complications. So the naive approach is on an Aries agent is everything's talking https.
Stephen Curran: you simply increase the instances and scale up by load. Stephen Curran: And so as as things get busier. You add another Aries instance, and another Aries instance, whether that be occupy or Aj or whatever it is. you've got agents out in the while talking through this endpoint, and then that gets load balanced across these. You also have the controller whoops Stephen Curran: didn't need to do that, but you also have the controller. that's sitting out there. Stephen Curran: hang on 1 s. I move that.
Stephen Curran: you also have the controller. that is talking to the various instances. Stephen Curran: so a big issue there, if the instances are lost. Messages are queued on these instances. If if things get really busy, the instances start to queue messages if the instances fail, or or, more precisely, when the instances fail, because it will happen. Cute messages are lost. So that's no good. The next approach that we've taken is, we add, a redis queue. Stephen Curran: the way it was done in the first cut of this Stephen Curran: group from keyboard first began this work, or a person from Quebec. And this work was to add a relay that basically received messages sucked them on the queue, and then Aries instance to stuff out on messages onto the queue, and those relays would send them back out. And this was all good when all we talked were https all of the ages the Controller could talk to the endpoints. Stephen Curran: Messages would go in and out of the queue. So Redis is just a sample of the queue. You could use capital or other other items other persistent queue mechanisms, for that. Messages are not lost. When Aries instances go away or when relays go away. Stephen Curran: Any instance can process any inbound message, any relay. You can send any outbound message?
Stephen Curran: as noted the relay is it required? The are these instances Stephen Curran: perform all of the tasks to push and pull messages from the queue, and basically everything is stateless, which is exactly what we want. Yay, Happily, there's no state involved. And so things that's sufficient. If Stephen Curran: all we have is a cps. So media air adds more complexity. So the mediator picture is slightly different. Stephen Curran: I I first of all, I left out the control, or just to make it simpler. we talked about this. We can configure the mediator to the auto. Accept that. I'm willing to mediate for anybody that asks, or you can even make it a plugin to, you know a Fj. Or occupy, or whatever you're using as your mediator. so that doesn't have to be external.
Stephen Curran: an external entity. So just make it simpler. I I got rid of the Controller. Stephen Curran: agents send in messages. Stephen Curran: that are sorry wallet. Send messages directly to agents. So a wallet wallets are over here. Agents are over here. Obviously, while it's our agents, too. They're all wages, but they the way the agents use? the mediator, is different from how the Wallace uses. So
Stephen Curran: while it send messages directly to Asians, so that means when our wallet needs to do an outbound message. It can send it directly. It does not have to flow through the mediator, and that's generally what's done. So messages do not flow through the mediator. When this happens. Stephen Curran: Inbound messages are all out down to a specific wallet after mediator instance processing, so the wallet may be offline. So it has to get queued. So we probably need Redis again. So this idea that we're sitting here with no queue. That's probably naive, so that we're going to have to address that. Stephen Curran: And then the next thing, those wallet to mediator instances are are sticking. So so what I'm saying, there is basically a shoot.
Stephen Curran: a a a message from an agent coming in is bound for a wallet, but it has to be sent through a mediator instance that interprets the message. Figure out the wall that it's for, and then sends it to that wallet. Stephen Curran: Now, what Stephen Curran: what happens is the wall. It's mediator instance connections are sticking. So that means when a message comes in after the connections been established between a mediator and a wallet, the next message has to flow outbound to the wallet through that mediator. So now, if the load Balancer receives a message, sends it to the media to the first mediator instance, it has to send it over to the second mediator instance Stephen Curran: and then over to the wallet. So that means basically, we're no longer stateless. Stephen Curran: And that's where the problems come in. Note that the connected instance, this mediator instance is talking to this wallet until one of them disconnects, and then the wallet has to recap. So this mediator could disappear. This wallet could stop for a while and then start again. When that happens, this wallet is going to go through the load balancer again and establish a connection to some
Stephen Curran: any of the mediators could be the same, one could be a different one. So that's where we also get into handling issues. Stephen Curran: I see there's a chat. Hope this is helpful. Stephen Curran: plus. He was just asking if Mobile to Mobile I mean, this wallet is talking to this wallet from each of the wallets. Perspective. They're just agents. They have no idea that they're they're each other's wallets, or they are our fellow wallets. So if a wallet is talking to another wallet, each of them perceives the other is just some external agent, and it doesn't care. Know that it's another wallet. It that's a that's a circumstance that I would Stephen Curran: a detailed, that it's not really relevant to this. Both of them have to do it. But it's not. It's not a special case. In other words, that's not really a special case.
Clecio Varjao: Oh, just on your second point where I said, Wallace sent a message directly to agents. I guess that's how what you're making. The distinction Stephen Curran: is in a conversation with another wallet. It thinks of it as just some other agent. It doesn't care it to the wallet, and it it might even, you know, it could be a wallet that doesn't share the same mediator, even Clecio Varjao: right so. But to generalize mobile wallets did not send a message directly. That's what Stephen Curran: they send it to, whatever the endpoint the other agent tells it. To Stephen Curran: whatever whatever endpoint this agent has said, if that's a mediator, that's a mediator, if it's directly to the agent, it doesn't matter. I could. I could say to what th this should say, I guess, directly to the ages endpoint to be more precise.
Stephen Curran: that makes sense. Clecio Varjao: not quite because in a wallet, to all that situation where they have here Clecio Varjao: did. Yes, you're right. It's always directed by the the peer that defines the endpoint. But I'm just making a point that outgoing message can also go through the mediator if directed. So Clecio Varjao: it's not a rule.
Stephen Curran: I don't know of any case where it would go through the mediator, unless. like, if the 2 wallets don't have the same mediator, it wouldn't go back through the same mediator. It would go to a different mediator. Clecio Varjao: Okay? we've figured out we can put that on the we can keep going. It's just like, is it not clear the mobile to mobile agent situation. there's 2, Clecio Varjao: I think, I think, as far as I understand, and maybe from my Fj. Or from from a I can. It's better. There's a communication through the mediator. Yeah. So basically, the the end point of the other agent happens to be a mobile agent.
Kim Ebert: And so it would also have a mediator, and it might be the same mediator as the other mobile agent is using. And so from mobile to mobile communication. The outbound communication may also go to the same mediator. Stephen Curran: But again, I I yeah, I I I mean I I that's totally true. What I'm saying is, it's I don't think it's relevant to the use case because Stephen Curran: the wallet sends it to wherever the aid the other other party says to send messages. It doesn't Stephen Curran: say to the mediator, Hey, I need you to send this message over to this other place for me.
Stephen Curran: it! It's sending it to where the Stephen Curran: other agent, whether it be a wall or not, tells it to. Colton Wolkins: and I think a bit of clarification here the mediator cluster that's being referred to in this diagram is the individual wallet's mediator. When it is communicating to an agent that agent may be, or maybe behind its own mediator, which is separate from the wallets. Mediator. Stephen Curran: Correct. Rodolfo Miranda: Stephanie. 1 one question regarding this the cluster. So that means that they all those instance mediator, have a share database or share storage, have all persist the messages, but also persist. The keys all share the keys.
Stephen Curran: Yes. Stephen Curran: yes, these I I took that detail out. But yes, there's a shared database here that all are using. So so as in the as in the previous diagram, there isn't, you know. I ask our or, you know. Stephen Curran: secure storage that is shared amongst them. Stephen Curran: And then with this, I'm adding. Stephen Curran: now we've got to have red us because we have to have Redis, so that Stephen Curran: messages for Wallace that are not connected, or for the loss of mediator instances. You have to have some sort of persistent queue. So you don't lose that Stephen Curran: again.
Stephen Curran: relay, is it necessarily required it could be a a mechanism. So again, a relay is simply just something that is, receives messages, puts them on the queue, takes things off the queue and sends them along. Stephen Curran: Oh, the the the other thing that I Stephen Curran: I have a new map. I have a new trap. The other thing that I didn't highlight Stephen Curran: is that Stephen Curran: inbound messages all come in via Https outbound message or or messages to wallets. between the media and the wall, up our web sockets. So these are persistent. There, there
Stephen Curran: they've done over Hdp, but the connection remains, and multiple messages go back and forth across them, and that's the stickiness that that is, that is there. So once a wallet. Stephen Curran: a step. And actually, I'll get into that in the next in the next slide. So so how do, how do ballots connect Stephen Curran: to relays? Stephen Curran: Is this Stephen Curran: level of hopefully, this level is is right. So the starting point is, the wallets are not connected to the mediator at all. they're just Stephen Curran: live. They activate. And then and they create a did call and request to send to the wallet. And that request is gonna be one of 3 things. One is, will you be my mediator? So the wallet has never connected to mediator and wants to have the mediator, be it Stephen Curran: I have to have that mediator instance be it's mediator, so will you be. My mediator is one type. The second one is, oh, I've got a new connection that I want to establish Will you mediate a new connection informally? So this is where it's it's establishing a peer-to-peer connection with some other agent in that agent could be a a, a another wallet, or anything.
Stephen Curran: and then finally. Stephen Curran: a wallet that perhaps just disconnected or just came online is gonna connect is going to create a request that says, Do you have any messages for me? Stephen Curran: So one of those 3 messages are gonna come in. They're gonna send the the the mediators to send that request Stephen Curran: the load balancer on the instances gonna route that to a receiver, whether that be a mediator, a relay. I'm just calling it a receiver. The wallet and the receiver from then on remain connected until something causes that connection to be lost, and while they're connected any messages between the entire mediator cluster and the wallet. Stephen Curran: all are all must be Stephen Curran: sent through that particular receiver. So coming back here, if this wallet connection with R. 3. Any messages destined for this wallet must be sent by R. 3, because it's got an established connection. Web socket connection R. 2 can't initiate a connection, and wall and the wallet. So therefore it can't send a message to this specific wallet. So that's that's the stickiness that's involved. Stephen Curran: If any new messages come in for the wallet to the mediator. So again, any any agent sends the messages in
Stephen Curran: to the to the mediator. it must be routed through whatever is connected to that wallet, you know. R. 3 or Stephen Curran: If there's a drop connection we start back with the. Do you have any messages for me? And that's likely to go to a different receiver. Stephen Curran: So we wind up with Stephen Curran: you know it gets load balance, it gets sent to a different receiver instance, and that receiver now has the connection. So the challenges you. You wind up with that
Stephen Curran: Each mediator instance is connected to a bunch of wallets. Stephen Curran: So each has a wallet id and each has a you know, some sort has a web socket connection. So one question is, how many web sockets could each instance handle? Stephen Curran: how many does it need to handle? So if you've got, you know, province of of British Columbia has 4 million residents that may get a B C wallet. How many? how many are going to be active at any one time we'll have an active web socket. And how many instances? will we need to handle that many web sockets distributed across?
Stephen Curran: a thousand 10,000. You know. What's the number? We don't really know. agent send in messages. Address to a wallet Id. So inbound messages coming to the mediator that are destined to a wallet. Stephen Curran: aren't they? Don't know the The 2 address of the wallet until the message has been processed by a mediator instance. So in the case of a relay, the relay will be able to interpret the message. A relay is just passing on messages inbound and outbound. So we're assuming it. It doesn't decrypt the message, and therefore it can't determine the destination. Wall, it did Stephen Curran: so. That has to be done after a mediator instance is processed it? this. This is the tricky part. How do the messages get to the right instance? given that there's Stephen Curran: If there's no connection it just goes into a pending queue to be picked up when the wallet connects. That should be a question mark that should be a statement.
Stephen Curran: if queue how it is an instance? Query the King, for all its messages. So suppose an instance has 10,000 web sockets, you know 10,000 wallets connected via web sockets. How does it find all the messages for all those 10,000 wallets? presuming it can't go polling through to say you have any messages for wallet? One. What about Wallet? 2. What about Wallet 3 can't do polling? It needs to pick it up somehow. Stephen Curran: What happens to all those cute messages when a wallet disconnects and then disconnects first of all. So now, whoever was handling that whichever instance was handling, it no longer is talking to that wallet, so it can't send it any more messages Stephen Curran: further. When the wallet reconnects, it may reconnect you know. A minute later or a half a minute later, because it just went out of cell range and then and then switched over to another. You know a a another network instance. Well, it's gonna reconnect, presumably to another instance. And now back instance has to find all those messages. Stephen Curran: And then, finally, what happens to the queue messages, for instance, shuts down. So an instance is handling the messages. It's it's sending messages along to the the wall. It's it's connected to it. Shut down. All of those messages have to be re-cued, and then or or at least picked up when when the wallets reconnect. Stephen Curran: So we certainly need to add state. So you know, we've got some sort of table of shared state here that says, Oh, well, you know what one did is currently connected to R, one using web socket, one wallet, 2 is using web socket, 2 on our one. That's a mistake. There. wallet 3 used to be connected to our 2, but then it disappears. So now it's not connected to anything
Stephen Curran: and so on. So some sort of state has to be managed somehow to enable those to pass. Stephen Curran: and with that I sort of leave it. Stephen Curran: that Stephen Curran: I don't have any more, and Ken's got a question or a comment. there's one more problem that would need to be addressed, and that is what happens if there's 2 web sockets for the same wallet connected Stephen Curran: there. You go? Didn't know that one. Kim Ebert: And there's a couple of different stories there. That could be handled. It's a more advanced use case. that we could table from now. But it's definitely worth keeping on the
Kim Ebert: on the Sam Curren (TelegramSam): Stephen you're still sharing. I don't know if you knew that. Stephen Curran: Steve, you're still Stephen Curran: there. You go? Sam Curren (TelegramSam): Rudolfo, your hands up.
Rodolfo Miranda: Yeah, Steven, how you still, instead of using web sockets. Rodolfo Miranda: they use https with the sort of Rodolfo Miranda: can can you hear me. I? Yes, we can have San Diego. Rodolfo Miranda: Okay, using https with it, polling mechanism, or maybe push notifications to go on and find them Rodolfo Miranda: the messages, because that that maybe you you're gonna buy the web. So that is messy Rodolfo Miranda: with me and soft wallets. Sam Curren (TelegramSam): we that is possible. Sam Curren (TelegramSam): although I Sam Curren (TelegramSam): We have a a good answer here in a second. So hold that thought.
Sam Curren (TelegramSam): I think so. I want to hear from from animal. But I happen to know that Stephen asked a bunch of questions, and I know that we had. There's some cool stuff that we would like to share and so I'd love to throw it over to I I don't Sam Curren (TelegramSam): either either Micah or Colton from the I don't know who is going to be driving. Micah Peltier: I think I will go ahead and start the conversation, and pass it off to Micah Peltier: Colton here in a moment. Micah Peltier: Let me figure out zoom screen share. can everybody see this? Sam Curren (TelegramSam): Okay, so cool.
Micah Peltier: so Micah Peltier: we are talking about a a repo that we have called socket deck Micah Peltier: and it's not Micah Peltier: public just yet, but we'll be making it public here. Micah Peltier: later. Today. I think we just have a couple of cleanup items Micah Peltier: before we do that but essentially it's Micah Peltier: it can be stood up in front of a a cloud based mediator. Micah Peltier: and what it does is Micah Peltier: it holds web sockets for Micah Peltier: arbitrary agents while it's Micah Peltier: and will forward web socket traffic across Http. Micah Peltier: you can set up arbitrarily many behind the load. Balancer
Micah Peltier: the web socket will be associated with a specific instance of socket that And so when when Alice, for example, sends the message Micah Peltier: up to another connection through the mediator. Micah Peltier: It'll go to this this first specific instance of socket, doc. Micah Peltier: and then sock and duck will sort of convert it into an Http message Micah Peltier: with enough metadata to associate Micah Peltier: that Micah Peltier: message. That connection with this specific instance of socket. Doc. so that when the cloud mediator needs to Micah Peltier: respond to Alice. it knows. Which instance of soccer do you send it to? Sam Curren (TelegramSam): So I'll highlight that there's no shared state between the socket block instances the state that's necessary is passed alongside the inbound message to the to whatever back end. It is a cloud mediator in this diagram. Sam Curren (TelegramSam): and then that State includes a back-end Api endpoint that is not run through the load balancer that links to the specific socket Doc instance, so that when a return message is sent. It knows exactly which instance to reach out, to, to pass it over the to the already or the the still connected web socket.
Sam Curren (TelegramSam): so no shared state between socket dock instances means a horizontally scalable without limitation at that level Micah Peltier: right? And and if Alice were to disconnect and reconnect. Micah Peltier: you know she wouldn't need to know which instance of, so that she was connected to initially. Micah Peltier: and so can Micah Peltier: the the load balancer can can reconnect with any instance of talking about Micah Peltier: exactly. and and that'll be fine on. Micah Peltier: And we we Micah Peltier: recently ran some tests. with this setup and all.
Colton Wolkins: What Before moving on there to the test. I do want to go ahead and add on that Colton Wolkins: when, because you mentioned when Alice disconnects Colton Wolkins: So when Alice disconnects the socket, Doc process. We'll go ahead and send off a message saying, Hey, Alice has disconnected Colton Wolkins: giving the mediator Colton Wolkins: a to potentially Colton Wolkins: start queuing the messages instead of trying to send like forward them on in live delivery mode.
Colton Wolkins: And Colton Wolkins: Steven, I see your hand up. Stephen Curran: The the question I have for coming back to that is, is messages coming from other agents going into the mediator. How do those get associated with Stephen Curran: the dock? Stephen Curran: The socket talk Sam Curren (TelegramSam): so they don't.
Stephen Curran: Yeah. So so you're asking the right question, socket Doc doesn't actually care. It does no evaluation of the messages. And so what matters is the software on the other side. Sam Curren (TelegramSam): So if you are, if you're using socket with an agent, and and you want to keep track of you know which which agent is connected, then, on an inbound message, the first inbound message from that agent you're going to record the the socket dock identifier which basically contains the Uri for a return message. Sam Curren (TelegramSam): and it, you're gonna associate that with whoever actually sent the Didcom message that you process socket. Doc doesn't do the processing. which means that if you then want to send a a message back out at any time, whether it's a live delivery mode or just a message that the mediator wants to send. Then it's capable of of looking that up in state in that, and and then when the you'll you'll see there's a post to message, Uri, and a post to disconnect Uri.
Sam Curren (TelegramSam): So when when the socket is disconnected, a a a message just with the disconnect you are I sent to the or or just with the the idea sent to the Cloud Mediator. Sam Curren (TelegramSam): so that the cloud Mediator, or whatever agent is behind it, is, is capable of cleaning up their state and knowing that that a current socket doesn't yet exist for that. Sam Curren (TelegramSam): And party Sam Curren (TelegramSam): does that make sense. So all the State is actually held by whatever agent is behind it. Socket Doc just handles the problem of holding on to the web socket and allowing for messages to be able to be sent back to the back to Alice in this diagrams case. via the the external host and port you know. So send Sam Curren (TelegramSam): you know. Send you all right. That's actually passed. Stephen Curran: So basically that Stephen Curran: diagram that I showed her that Stephen Curran: trivial table that I just just made it up is basically gonna consist of a wallet did, and a URL.
Sam Curren (TelegramSam): Yes, so we use it as an id. We get the URL and and the URL is Stephen Curran: go ahead. Sorry. Stephen Curran: and here is their current. Stephen Curran: URL to send them to whenever a message comes in. That's correct.
Stephen Curran: Okay. yep. Sam Curren (TelegramSam): So the socket Doc is is is as simple as possible. We say cloud, mediator, or other agent, because Socket Doc doesn't actually care. What is important is that if you have an agent that is receiving messages, it needs to understand the format being sent because it's not just a did come message. It has a little bit of metadata alongside it which contains that that URL which services id for the socket and so there, there may be a little bit of of stuff put in front of it to make that happen. Sam Curren (TelegramSam): there's the good news is, and that and there's code here on the screen. And again, this will be open later today. so this is what the message actually looks like. There's a there's a metadata that has a struct in it with the call back uris, and then and then the and then the actual messages passed alongside there as well. Sam Curren (TelegramSam): And so this isn't a lot of code, but it performs a really important role that allows for a an abstraction to occur architecturally speaking, between the maintenance of sockets and the and then a back end processing engine that doesn't have to be capable of sockets at all. That's to have a minor understanding of how to use socket. Do as it's front end, if you will. but that's pretty minor Sam Curren (TelegramSam): and and everything else. So we've we've had this, and we've used it But we wanted to verify that it performed the way that we hoped it would, and
Sam Curren (TelegramSam): we ran some performance tests Colton Wolkins: Yup, and and just to clarify like in this scenario here, where you now have the wallets connecting and through socket, Doc. Colton Wolkins: the wallets. As far as they're concerned, they're still talking to the mediator via web sockets. But now, as the media, as far as the mediator is concerned, it is only communicating via Http get or Colton Wolkins: Acd post requests inbound and outbound John Jordan: Now moving on to the performance. Metrics that say, I mentioned, if one of the instances of socket doc fails.
John Jordan: what happens? Kim Ebert: If it fails, then all of those agents will need to reconnect there by associating Kim Ebert: their new web socket with the new Kim Ebert: a mapping inside the cloud mediator. John Jordan: Right? So. But their state, I guess, is held by the mediator. So John Jordan: no problem. Colton Wolkins: then the posts would fail right, the posts would fail. So if the socket Doc came back up and we tried to post a message. But Alice was no longer associated with this socket. Doc. Instance, here Colton Wolkins: the post to socket, Doc would fail. saying, Hey, we don't have that guy connected to us
Colton Wolkins: which point the cloud mediator would say, Oh, hey! My post failed. I need to queue that message instead John Jordan: and wait for her or you to connect. Sam Curren (TelegramSam): Yep. John Jordan: okay. Colton Wolkins: So we we ran a few tests. Just kind of scaling up the system this test here Colton Wolkins: we ran it with 3,000 users.
Colton Wolkins: just to see how it would perform. And we were getting just you know. Colton Wolkins: the user. All these users are just Colton Wolkins: establishing a connection to the mediator and then doing a ping, a continual ping every about minute or 2 Colton Wolkins: to the mediator, and we noticed that there weren't really any failures. Colton Wolkins: and everything was going smoothly. So we changed from her, and the system Colton Wolkins: use on both the nodes was fairly minimal.
Kim Ebert: So what you're indicating here, Colton, is that we have 2 socket Doc nodes in front of the cloud mediator of some type. Colton Wolkins: Yes, and actually. So what we're doing here Colton Wolkins: we have the Amazon aws load balancer in front of 2 Colton Wolkins: t 2 Colton Wolkins: micro nodes which have 1 GB of RAM.
Colton Wolkins: And so we, we're currently just doing using. They're performing these tests with these 2 nodes. And then we have a cloud mediator behind that right? Okay? Colton Wolkins: And so fairly minimal RAM usage and CPU usage here throughout that test. Colton Wolkins: Next up, we decided to bump up the numbers by quite a bit.
Colton Wolkins: and we bumped up the number of users to Colton Wolkins: 10,000 users Colton Wolkins: the the drop here is because we were Kim Ebert: so. I actually think that this is a statistics issue with a 95 percentile to take into account the pings. the 95 percentile here includes the Aj Startup time. Kim Ebert: but as soon as you have enough pings that may cause the 95% tile to drop lower there. Kim Ebert: and so that's new, just to Kim Ebert: a statistic. a function. Colton Wolkins: I I also thought that the drop off was due to the fact that we were starting up those Colton Wolkins: Afj agents or wallets in this case.
Colton Wolkins: we slowed down the spin up just due to a limitation on the Kim Ebert: best machine of the test machine itself that was sending these requests out. was due to a CPU limitation. Kim Ebert: But again, it's a statistic issue. If you look up above at the percent tiles Kim Ebert: inside of the different functions here. you'll see that the ping stayed relatively consistent. and the on start time was relatively consistent. If you look at the different percentiles here.
Colton Wolkins: And the startup Colton Wolkins: was dragging that percentile way up compared to where the ping. Colton Wolkins: And again, like 10,000 users. Colton Wolkins: no big deal. We're still going ahead and handling everything, all those socket communications Colton Wolkins: being load balanced.
Colton Wolkins: And we're still. Colton Wolkins: We're only at 436 MB RAM usage on these 2 nodes. CPU usage is still relatively small. Colton Wolkins: so we decided to bump it up even further to Colton Wolkins: 12,500 simultaneously connected users over web sockets. Colton Wolkins: all of those ping still. Colton Wolkins: for all intents and purposes, chugging along just fine across these 2 Colton Wolkins: load balanced T. 2 micro instances on Amazon. and Colton Wolkins: our RAM usage is still below 500 MB on both of these 2 nodes, and our CPU usage Colton Wolkins: with these active pings, is still relatively small.
Colton Wolkins: And so what this means is that Colton Wolkins: all of these agents, they are all of these agents that we have connected, and running these pings. Colton Wolkins: They are Colton Wolkins: live active agents, in a sense, just waiting for messages. Colton Wolkins: and so, if they were mobile agents, this would be like 12,500 people all at once, just opening the app and having it open on their screen. As soon as they would close the app, it would
Colton Wolkins: close that web socket communication. Colton Wolkins: but Colton Wolkins: our tests here are without that whole close of the app. These are all active web socket connections. Sam Curren (TelegramSam): So Charles got to begin a question why the focus on web sockets? It shouldn't be transport agnostic. A mediator could use smtp the the preferred. The reason why it's preferred for mobile agents is that you want a message to arrive really soon without necessarily pulling. You can also rely on on push notifications for this.
Sam Curren (TelegramSam): but when you're if the if the app is open, the likelihood that the person is going to be receiving a message goes up substantially as opposed to any other time, which means that maintaining a live web socket connection produces a really really fast user experience for inbound messages. Sam Curren (TelegramSam): but we've had a challenge up to this point about scaling the number of web sockets that you can maintain easily. so these t 2 macro instances a cost 10 bucks a month on. Aws, I I've realized on different platforms that will be different. But just to give you kind of a sense of Sam Curren (TelegramSam): what the cost is. Sam Curren (TelegramSam): And and we're making an educated guess that that at this point we can support 10 1,000 Sam Curren (TelegramSam): sockets per t, 2 micro instance given this behavior, obviously, some, some more real world experience will be helpful in and evaluating that number. But we, we feel like this is a really great way to cost effectively provide that that connected web socket experience. Kim. Kim Ebert: I did want to mention that. the
Kim Ebert: low testing could go much higher. It was a limitation of the low testing configuration at the time. not the micro instances or the cloud mediator. Sam Curren (TelegramSam): Yes, totally. We. We had a single instance driving this and we have the capability of doing clustered test driving boxes. And that's what we'll need to do to to drive it above that we were running a very large instance to to run this number of sockets. and and what was happening, and that test, by the way Kim mentioned this was using afg underneath. Sam Curren (TelegramSam): So unfortunately, we have, like 1 min left. this has been fantastic. but hopefully. So this falls Steven, into your your relay in the in the diagrams that you were showing and can and can perform that relay really really well, and it doesn't care what's behind it. It can actually feed those messages, of course, into into a queuing system, or any other mechanism that would be useful for that. Sam Curren (TelegramSam): and we also have a proxy mediator that that that that Kim just dropped a link to that allows. This is what we use to avoid the the Sam Curren (TelegramSam): oh, shoot! Sam Curren (TelegramSam): I just blinked on this. Oh, you can see this openly later today. We almost had it open. it! I think it would work with as the mediator as long as Afj was able to process the inbound posts, so there may be a tiny shim in front of that of that teamo in order to use afj as that. the, there's no, there's no pre processing of the Didcom message is done. It's just that we're posting that with some additional metadata. Sam Curren (TelegramSam): so really quick Sam Curren (TelegramSam): in the agenda. there is a link that I that I need to share, because we need review on it. Timo has produced a a a legacy. Did transformation document that needs review and so that we can use this.
Sam Curren (TelegramSam): the proposals that we use this as a way of of transitioning off of the legacy. Did given sort of what was pioneered inside of a J, and so teamo created this, this does need review and we could bring it up and talk about it online. And in our next meeting. but I wanted to highlight that as well. we will be posting the socket, Doc Sam Curren (TelegramSam): repo link inside of the the area channels on discord. and for your perusal. And we would love some questions. Feedback bug issues all of those other things. if the community is acceptable to it. We would. We can transfer that repo into the Hyper Ledger organization. it'll first be made public as a as an in Dco repository. with the license file, etc., in there. But But we we would love to transfer Sam Curren (TelegramSam): for that. We just want to make sure that it's desirable by the community to do so. Sam Curren (TelegramSam): and I apologize for going over But thank you. And we can talk about this, and then pre, please reach out Colton and and Micah are the are the experts on this code, and and they'd be happy to to answer any questions. They are also on discord. Sam Curren (TelegramSam): any any final comments, since we're 2 min over.
John Jordan: Great stuff. Thank you. Stephen Curran: Fantastic, awesome work. Stephen Curran: you know.
Timo Glastra: Thanks. Everyone.
2023-06-28 17:10
