Hello everyone, and thank you for joining us today. My name is dustin person. And i'm joined today with my colleague nate molibai. And we'll be talking about unlocking, iot data for research, in healthcare. To start with i'll go over the brief session outline, so we're going to cover. Iot, in healthcare. Go through some of the challenges, you might face if you're using iot. With healthcare data. And then we're going to cover. What we have done, so far in the health cloud and data team. To solve these problems. And then, later nate's going to be going over several examples, on how you can use. Uh iot, data and some of the technologies, we developed. Through an end-to-end solution. So to start with. What is iot. In healthcare, or as we call it iomt. Well imt, stands for internet of medical things, so you can think of this as a subset of medical devices. That. Deal with, specifically, patient data. They can come in many shapes and sizes, they can be worn, so a lot of you be familiar, with. Devices, like fitbit, and apple watch. The, person. Wears all the time, and, they are passively. Monitoring, things that the, person. Does and vital signs. There's also a class of devices that are ambient, so you can think of sensors, that exist in say like a hospital bed. Or. Monitor a room. And. They either. Take medical readings, or vital signs, or. Report on patient, behavior. And then there's a class of devices, that can be, used, so they'll be like injectables. Pills and things like that. So what are some of the key scenarios. So. We see them as. Kind of three pillars. The first would be patient monitoring. So we use sensors, and pros, which stand for patient reported, outcomes. Which are surveys, that a patient. Will fill out to add subjective, feedback, to any of the sensor data. And this is, valuable, in a lot of remote, care. Telehealth. And. Chronic disease management scenarios. Another pillar that we see is in. Research, and life sciences. Specifically, around clinical, trials. And lab data analytics. And then third. There's the whole category, of smart hospitals. We're not going to really, focus on that. For this presentation. But just want to make it aware that it's, out there. Another thing to call out is the use of iot. And iomt, devices within, healthcare, is growing rapidly. It's projected, to reach. 340. Billion dollars, by the end of 2025.. So it's a. Very exciting and growing space. So, with that brief introduction. What are some of the challenges. Of implementing, an iomt, solution. So if you're going to be building, your own imt, solution, there's several, things that you, may, need to account for. The first is, ingestion. Of high frequency, data. Not all devices, are high frequency, you know some of them run, may only be, reading, you know once a day. But there's also, a lot that can, produce data, at very high rates you know, every second or sub-second. So. If you want to be able to interact with that class of devices, you would need to account for that. You also have, a, desire, attention, for, low latency. So, you want to be able to get or ingest the data and make it available. As, soon as possible, as you can. You also have to associate, data. Between. From the device, to a patient record. To make it meaningful. There's also. A multitude, of devices, out in the ecosystem. There's, not a lot of standardization. Out there. And. They all, report. Data in their own unique way. Another. Key thing that you would need to account for is interoperability. Specifically, within, the. Healthcare, data, realm. And specifically. Here this is where we're targeting. Fire which is a health open healthcare standard. That's gaining traction, in the community. And. Provides, us a, way of, sharing data. Across, hospital systems.
You Also have to be aware of privacy and security. And account for those. And. Also what do you do when you need to update the data that's coming from the device. So. With those, uh key points, uh. Gone through, uh, what are some of the challenges, that you okay, especially if you're building your own solution. So one of the key ones, is late arriving data. Especially. If you have a wearable, device, that's. Using some kind of. Gateway for internet connection. Like a like a smartphone. You can have data that's, being collected. You know for hours or even days at a time and then once. Internet connectivity, is established. That data will all be. Sent and transmitted, so you need to be able to one. Be able to handle that burst of data. And then. Make sense of it. Appropriately. And associate, with the right time period. Similarly, to that um. There's really no standard, in how the device, may send data, so if it's been offline, it may send, the most recent data first. And, and, work back chronologically, or it could send out, the oldest data. And, there's really no particular, order. So, whatever solution, you have would need to account for that. Uh, there's also the problem of duplicates. Because. The device itself, may choose to resend data. Because, it was in the middle of transmitting. Connection was lost and it, didn't get the acknowledgement, that a, particular piece of data was successfully. Received. You have to be able to handle linking the data to the patient. And there's this uh. Tension between, balancing, latency, and load on the system, so. You know like i mentioned earlier you want to get the data available. As quickly as possible. But. If you go with near, real-time, or real-time processing. That. Adds, significant. Challenges. On the back end. There's, also. As i mentioned earlier there's devices, that. Run the gamut of how they send or represent their data streams, there's really no clear standardization. And, um, this, particular, last one is managing the number of resources, created, in fire, so. Resources. You can think of them as like entities. Specifically, in fire we call them observations, when they're we're dealing with, health data. Specific. To a patient. So you can imagine, is if i have a device that's streaming, health or heart rate data every second. And i'm creating a distinct, observation, for each of those. That can present, some some challenges, and how to enumerate, or. Retrieve, that data. Because even. Pulling back. Like a patient's. Day's worth of, data. Percent, several thousands. Of, records. So. We've had. Through the years several. Prototypes. And partners, that we've. Been working with as we come tried to come up with solutions. And wanted to go over some of the lessons, learned. And kind of what we have used, um. As. Guiding principles. When we, built our solution. So one of the things as i just alluded to in the last slide is that a single value observations, are not sufficient.
So We don't want to just store. Or, have only the option of storing. Um. That heart rate data as a single point. Thankfully, fire, does have a, time series format, called. Sample data, that allows us to essentially, group. Collections, of data. Into a time series format. So. We, one of the things that we do here is you define, an observation. That say could be for an hour. And then every single measurement, within that hour for that, say heart rate, would be represented, within that observation. The. When we're looking at prototypes, or how we might tackle this problem. One of the first places we went to was on the device itself. You know is what could we put on the device. You know how could we standardize, and have. Basically. Some code that would run on the device that would. Handle. Uh the recording of the measurements. And. Sending them to the cloud. That presented, several challenges. And, we opted to actually try to make, the solution. Device diagnostic. And essentially. Allow. Us to receive the data in. Any way possible, on the cloud side and reconcile, it there. Some of the, problems are you know device, complexity. That they, run in in the gamut of signals some have a lot of processing, power some have very little. And also if you're doing all the processing, on the device itself. That inherently. Introduces. Latency. You know the device manufacturer. Or, the. Person that writes the gateway. You know may have, a buffering, period. Which is which is fine if that's what they choose to implement but we didn't want to have, an inherent, solution. That um. Had latency, built in. And then code portability. Um you know that, each, device, runs, on you know its own particular, os, or, firmware, version. So making, a one size fits all. Solution, that, lived on the device was problematic. Another thing is fire is an evolving, standard. And. One of the things we didn't want to do, is if we were doing. The conversion, and collection. Into fire, on the device, itself. Then we have to. Ensure that you know if we want to upgrade. The version of fire. Um. That but then we have to make sure all the devices, are upgraded, as well where if we do that in the cloud. We can kind of decouple, those things. The other, um, interesting tension point is. Specifically, when we get into these um. Value sample data. Um. Time series formats, the. Fire payloads themselves can be very large. And we wanted to kind of keep, the the footprint, to a very small size because some of these devices. You know. Are. Very tiny. So that brings us to kind of, two solutions, that we've built. So we have both in, oss. Version, the imt fire connector for, azure. And, a. Paths option currently in, public preview the azure iot connector for fire.
The Imt, fire connector for azure, allows, you full control over the code so, if you want to take the, the basic, building blocks and then extend, it. Or add enhancements, to support your specific scenario. You're free to do that. But if you'd like to, have, a one-stop, shop, and, use. The solution. That we built as is, then i encourage you to look at the azure iot. Connector for fire. So. What is the. Imt, fire connector for azure. So. We provide an endpoint, that can process high-frequency. Imt data. Process payloads, with mixed data types. And we accomplish this by first normalizing, data. So we take data that's in a non-fire, format. And, merge it into a common intermediate, format. Then we group, that data. By. Various. Properties. And then we do the transform. Step where we convert it into fire observations. And finally, it's persisted. In the target fire server. So we're going to go through. These steps a little bit in more detail, to kind of highlight. What's going on there, so in the normalization. Step. We take these arbitrary, device payloads. Represented, here with you know. H rate heart rate in hr, so. Think of these as three different devices, that are own singing, sending their own discrete payload. To the cloud. With that data. As it's coming in we'll normalize, it into, a common model. Where and the output would be, based off the simple example. Heart rate 65. Heart rate 59, heart rate 88.. Another, important. Thing that we can we do in normalization. Is projection. So. A lot of times devices, will send up essentially. One message with various, attributes. Um. For different vital signs. This is great from the devices, standpoint, because. Uh you know they're usually sampling, these. Um, at the same frequency. And. Sending one message, is is convenient, and practical, for them. But, on, the, processing, side, and storage side. You. A lot of time your heart rate doesn't really have any direct correlation, to your steps. Or your, heart rate variability. And you'd want to store them in separate observations. And fire. So. To accomplish, that. Our normalization, step, also supports projection. So, you can configure, it such that these would get split out and then. Conversely, they could be. Stored as separate observations. In fire once we get to that step. In the grouping, and buffering. Stage. We have data, grouped according to the type so think of like this is like a semantic, type, that the. Person configuring, the system defines. Heart rate. Step count. Blood pressure etc. Window time, and, device. This is this essentially controls, your end to end, latency. And it's one of the options, in the open source that you can, tweak that's configurable, it's currently, locked into 15 minutes, in the public preview. But it's going to be something that we're going to open up as configurable.
In The future. So this, essentially, controls, how often, data is egressed. From the connector, into fire. So. If you tweak it lower, you're going to. Get data into fire sooner. But you're potentially, going to increase, the, load on your fire server so depending on your use case. You, you may. You know, want to have it uh very small, because you need to do some real-time. Analytics, real-time, processing. Um, or, if you, are, doing something that's more. Analytical. You may set a higher value, because you want to have the the system, it doesn't matter. If it's, on the minute available on the fire server. So. Finally, the the fire conversion. We have, several options. So, you can define the period. So this is if you're, using, the time series format that i mentioned. So you can say i want this observation, for heart rate to be bucketed, by say an hour, so every hour. As data is streamed in, or. Uploaded to the connector we'll figure out. What hour, time period, that data belongs, to. And. Merge it into that observation. We also support, various coding so this is where you would define, your link, snomed, or other code systems. For annotating, your data. You also can set the the data type. This is where if you were using the. Sample data type for time series i mentioned, or, if you have scenarios. For. String data. Or value quantity. Because the measurements. Are infrequent, you can configure that there. We handle, for you mapping the data to the correct observation, based off the time that it was recorded, as being. Observed. And we also make a deterministic. Identifier. So, as data, comes in. We can save it and then if, data for that same observation. That needs to be amended, later. We can identify, it and update it accordingly. And then all the observations, that we create are linked to both the patient and the device. And for those that are familiar with fire we support. Not just single value observations, but there's a concept of components. An example of this would be, you have a blood pressure. Observation. And you can have components, for diastolic, and systolic. Blood. Pressure. And. Here's a quick example. Of. Fire observation. That would be created, by the connector. So we have here like the internal id, and the resource, type. We have the, device.
And Subject, references. Which is the patient reference in this case. Coding, so right here we have a link code for heart rate. Uh the effective period. And this one is just a simple value, quantity. And real quick i'm going to go through. Some of the configuration, steps, so one the normalization, step is we have a concept, of device mapping. So, what this may look like here is i have a, sample payload from a device. We have a multitude. Of, different signals that are being recorded. And some properties. For. The. The date, and time that the measurement was recorded. And the device. Id. Your responsibility, for using the system is to configure, a template. That. Will. Know how to identify, and match this. So some key. Pieces here is we have a template that's supposed to. Map this to a semantic type of heart rate. And then we. Achieve this by this type match expression, here, so basically if this evaluates, to true then we have identified, it as a. Message of this type, and we'll proceed then to extract the various values defined, here. And then the output of this is a normalized. Value. On the fire mapping side. We'll take a collection, of normalized, data, like this. Combine, it with a template where we're mapping based off of the semantic, type that the, configure, of the system defines. And then use. Some other properties. To say. In this case we want the period interval. A 0 means it's a single instance so we're just going to create this, as is. The codes we want to associate, with it. And then, how we extract the values, and how we represent them in fire. And, this would be the output. So. With that, you may be wondering, exactly how can you use the. The iot connector. As part of a larger. System. So here's a kind of reference architecture, how we see the connector being used. So, on the left hand side we have the. Ingestion. Where we have. You know data coming from devices, that can either be, using, a device gateway. Using one of our. Azure iot. Products, like iot central or iot, hub. Or going through, a phone gateway, and connecting to, the iot connector directly. The, iot connector, then. Funnels data into. Our managed. Azure api for fire. Which is the fire server. That you can use to access the data that as it's been. Streamed in. You can also use this to. Ingest data from ehrs. And third parties into the azure api for fire and once there. You can, use it in various, applications. Similarly, if you wanted to do. Analytics. Once it's in the azure api for fire we support export. And anonymization. And then you can take it into. Various. Azure services, from there. With that i'm going to hand it over to, nate to go into, the demo. Okay thanks dustin. And thanks everyone for attending, so the demo i'm going to show today, is an end-to-end, pipeline. Using the azure iot connector for fire. And it was really easy to set up i mean it basically didn't require writing any additional code, just configuring, the different components. So. The first thing i'm going to do is take a minute to walk through some of the components in this demo.
I've Deployed an instance of azure api, for fire which is what we use for our persistence, layer. So azure api for fire, meets hipaa regulatory. Requirements. And is also iso. Certified, so it can be used to safely and securely, store, protected health information. Within the azure api for fire control plane. I enabled, an azure iot connector for fire. And configured, the mappings, to process, the device data that we're going to be sending, so dustin talked about those device mappings. That are used to transform, an arbitrary, json payload to fire, and we'll be taking a deep dive, into how this conversion, works, with data from an actual device. So. I have a device here. I'll hold it with my camera, it's. An eye health device. Made for measuring blood oxygen saturation. It's just an off-the-shelf, device i think it was about 40 bucks. And, i'm also. If you look to the right side of my screen. I'm projecting, my iphone. And. You can see i installed the ihealth app it's the orange, icon labeled eye health. And when i take a measurement, using, this device. The eye health app will write the blood oxygen saturation, measurement to the health kit. So i just want to take a minute to talk about health kit health kit is available on all the iphones. It's developed by apple and built into the os. And applications, and devices, can store, health data. On the iphone, using health kit, and this data can be shared securely, with third-party, applications. If and when a user grants permission, to share the data. So our team leveraged this capability. And developed an open source swift library, called health kit on fire. So health kit on fire. Exports, data from health kit automatically. To the iot connector. This library is available on github, along with documentation. A quick start guide, and a sample, app, that can be deployed in an iphone for testing and evaluation. And for this demo. You can see that i deployed. The sample app it's the white app icon labeled iot, fire. In pre-demo, i configured, the iot connector endpoint. So it would send data, to, an iot connector that i created. And also. I created a patient, in the azure api for fire that represents me. Now when the eye health. Application. Saves the blood oxygen saturation, measurement, to health kit, it's immediately, exported to the iot, connector, processed. And saved as an observation, resource. Into the azure api for fire. So. Once the data is there we can then use, the power bi fire connector to create a report. And, here's what that report looks like. So i created a kind of a quick and dirty dashboard, that simulates, a scenario. Where a clinician, is monitoring, blood oxygen saturation. For a group of patients. This first page shows. The measurements, taken today. And i'm down at the bottom. My row is highlighted yellow. Yellow means that the measurement has not been taken for today. And. Uh highlighted. Red means that the measurement is under 95. Uh. As is the case with grace owens here.
So I do want to call out with that with the exception to myself. All of the patients on this list are fictitious, and their data is all synthetic. So let's take a, quick measurement and see what happens here. So the first thing i need to do. Is i need to launch the ihealth application. So i'll launch that and then i turn on. The device that reads blood oxygen saturation. So when it connects to bluetooth. What happens is the phone starts reading data, and when i take the device off my finger it's going to create a single blood oxygen saturation, measurement. That's going to be written into health kit. So i pulled that off it says. 98. My pulse rate is 128, that's pretty high. Um. So. The health kit on fire app sees the new data immediately, and uploads it to the iot connector, and this is where the normalization. Process begins. It extracts the device identifier. The time the measurement, occurred, and the measurement data itself. At this point i'm going to take a minute to go under the hood and see how this data is being, processed. So, this is what the payload. From healthcare, on fire looks like. The measurement. Is here at the top. The time stamp, and device. Id are here. And. Dustin briefly, talked about, the two mapping, files that help transform, arbitrary. Json payload, into. Fire. On the right is an example, of the first mapping, used, to normalize, json data into the format that's easier for the iot, connector, to work with. So we use this json path expression. To determine, if this mapping should be applied to the incoming payload. And we see that there's a match so we can start extracting. The other values. That we're going to use to create the fire observation. We use the timestamp, expression. To locate the and extract the timestamp. So we know the date and time the measurement took place. We use the device, id, expression. To locate, and extract the device id so we can provide a reference. In the observation, resource. To the device that generated the measurement. And then we use the value expression. To extract the measurement value. Once all the, values have been extracted, we create a normalized, data object that is used internally, by the iot connector. And, this gives us the ability to accept, json payloads from many different devices, that may have many different data formats. So, now that the. Incoming data has been normalized, we could group, the data if needed in this example we don't need to because it's a single measurement, but, grouping is useful, if the data is expected to be streaming, or high frequency. And uh it's useful because we can as, dustin mentioned earlier, we can limit the number of observations, that are, created. So now we take the normalized, data and we create a fire observation. So i'm going to jump in, and show how that's done. So this, is the example of the second mapping json. The fire mapping. The fire. Mapping. Also uses, uh jsonpath, to locate, and extract, data so the first thing we do is check if this fire mapping should be applied to the normalized, data. That was created, in the first transformation, step. This, fire mapping should be applied to the oxygen, saturation, type. And we can see that this normalized, data, is of type oxygen saturation. And because the normalized data type matches we can start the process, of creating, an observation. So. We, establish. The value type that will be included, in the observation. And as you can see, this observation, will contain a value quantity. So value quantity can be used for single measurements, like what we're showing in this demo but there are other types i think dustin mentioned about. Valued sample data. Those can be used for streaming, and then, string values, and, also codeable, types, so, we support more than just, quantity types. So we locate the value. By name, in the normalized, data, in this case oxygen saturation. And then we write it to the observation, resource. And as you can see because the observation, value is a quantity, the value, is converted, from a string to a number. If this normalized, payload that contained a number it would just write the number as is. So then we copy the system, unit and code. From the fire mapping to the observation, so that the value is coded properly. And then any of the the codes in the template are copied to the observation. Here, we just have the one code but you can add as many codes that are required, to support, your business case. And then we write the time that the measurement was taken so we use that time stamp. To create an effective period start and end uh because it's a single measurement the start and end are exactly the same. Okay then the last step in creating this observation, is setting the references, to the patient. Uh that took the measurement, and the device. Uh the patient, sorry that the measurement was taken from and the device. That took the measurement. So this is done by fetching the device resource. Um, using the device id from the azure api for fire, now the health kit on fire application.
Or The health kit on fire sample application, will actually. Create devices, for you and fire. And that way the data that comes in, at a later date will be linked to it. So the fire id, is then copied. From the observation, as a reference. And the patient reference, that's been taken. Uh included in the device, resource, is copied to the, observation, so i want to call this out specifically. The iot. Connector, in this instance has been, configured, to look up the device. And a patient when the data arrives. So this means there's an expectation, that a device resource. With a matching identifier, exists. And that that device, resource. Has a reference to the patient resource. The iot connector can also be configured to create device and patient resources. Automatically. When data is sent to the connector. And that's just by configuring, it to do. Create. So now that the observation, has been created we can persist the data in fire. Okay, so now we have three resource types in azure api for fire we have the patient resource, which represents, me. The device, resource, which the, health kit on fire sample have created and then an observation, which was created, based off of the. Data generated, from this blood oxygen, saturation, monitor. For this demo, we're just going to import the two types to power bi. To gener, generate our report, it's going to be patient and observation. And then we can shape this data. Uh in this case i'm going to merge the list of observations. To the patients that they belong to, and that way i can chart the measurements over time. And then once that data has been shaped i can, create my visualizations. So let's jump back to the. The dashboard, and we can, see what it looks like, so i'll refresh this really quick. It might take. A little bit of time because i have, several hundred observations, now and then. I think we have ten patients. Once the date is done refreshing. It should update the ui, and you can see that the. 98. Value let's make sure that's correct yep 98. From the eye health application. Is properly. Reflected, here, i can also go through, and, drill through. So i can look at these other measurements. That i've been taking since. The. What is this the 18th of october. And so this, just maps out um all of my daily measurements, that i've taken, there's a couple low spots in here. But for the most part my trend line is going the right direction it's going up so that's good. And if we go back. To, the first, dashboard, here we can actually drill into grace owens and we can see what's happening here. So, here, grace owens again my fictitious. Patient. Her last measurement, was. 91. And it does seem like the trend line seems like she's trending down so it might be time to give her a call and have her come into the clinic. Okay so i did want to leave you guys with, some links to um. Some of the. Tools that we use today. Again this was a, no code solution i just literally deployed, the sample application, on the health kit on fire. Repository. I set up the power bi, fire connector which is generally, available and also. Created, an instance of azure iot connector for fire, along with the. Along with the fire server. And, so here's some links to the documentation. Okay, thank you everyone. Uh if you have questions. Uh, the. Contact information, was at the beginning of the conference. Of the presentation. So feel free to reach out to us directly. If you have further questions. Uh thank you i hope everyone has a great. Day.
2020-11-10