Unveiling the Secrets of Event-Driven Architecture and Technologies

Show video

Welcome to the Joshua Matos show your  place for diving deep into the vast   field of software engineering I'm Joshua  Matos your full-stack software engineer   and I'm here to make sure you become  the most awesome software engineer today's episode we'll be exploring event  driven architecture event dispatching   spring cloud dataflow and advanced message  queuing protocol we'll touch on some of the   cloud services you can use too so what is event  driven architecture why should we care about it   and most excitingly what can we achieve with  it suppose you didn't know how to leverage   these powerful Tools in that case you can  transform your organization by enabling   them to implement microservices event driven  architecture and most importantly allow your   teams to make real-time data-driven decisions  knowing about event-driven architecture and   how to leverage message broker systems will make  you an invaluable asset as a software engineer   event driven architecture Eda is a software  design Paradigm centered on events driving   communication for instance an item purchased at a  counter triggers an event to update the inventory   is design really emphasizes the production  detection and consumption of events events are   triggered communication is made decouple Services  collaborate and the systems rely on eventual   consistency we'll talk a little bit more about  eventual consistency later Eda decouples services   and enables asynchronous communication which  makes your environment scalable this decoupling   mechanism alleviates bottlenecks your application  doesn't have to wait for a message to be received   applications continue with their workload while  sending information elsewhere to be processed   imagine if the customer had to wait for the  inventory item to be updated before being able   to make the purchase applications become more  resilient to failure as Services can operate   independently if a service encounters an issue  the rest of your system will continue to run   this spot tolerant nature ensures the systems  remain functional delivering consistent and   uninterrupted Services event-driven applications  core strength lies in its ability to build systems   that respond to events real time by proactively  reacting to events applications can provide   instantaneous updates real-time notifications  and dynamic responses enhancing user experience   and driving data decision making it's worth  noting that spring boot uses concepts of event   dispatching to enable the publication of events  to other services this powerful pattern can be   implemented with or without a message broker  system streaming is another technique within   Eda enables the processing of events as they  occur such as financial data or sensor data   streams can be used for filtering data data  enrichment and other Technologies like machine   learning in an event-driven architecture separate  microservices may be responsible for their part of   the domain it could be an inventory equipment or a  user domain and an Eda these systems would either   publish or consume events the messages can be sent  to All Systems only one or several and this type   of architecture communication is asynchronous  events May not always arrive simultaneously   your customer may purchase an item that sends an  event that updates the inventory and cart system   these messages will not necessarily arrive  at the same time which is otherwise known   as eventual consistency Eda is powerful  but not ubiquitous so you may have to move   your organization or software development  team toward an event driven architecture   one issue you may encounter when introducing  an event-driven architecture into your company   is that the current code base is monolithic  meaning the applications are tightly coupled   the front end and back end services are tied  closely often in the same repository monoliths   are usually deployed as a single individual unit  shifting to an event-driven architecture creates   a more resilient and scalable infrastructure the  first step often involves moving to microservices   simply separating the back and front  ends can most easily accomplish this   doing something as simple as this can help shift  your organization mindset on designing and coding   software another approach is to create a new  service independent of the application does   the application require a new inventory service  create a separate backend only that responds to   inventories as time passes you can introduce new  technologies that facilitate your microservices   and enables event-driven architecture let's talk  about spring boot's event dispatching and how to   use it to improve your software development  events are a tool that allow different parts   of your application to communicate with each other  and help ensure your application remains decoupled   often when one part of your application needs  to collaborate with another part you'll inject   one class into another and then call one  of its methods or an API is introduced into   an application used throughout the code base  essentially coupling your application tightly   and spring boot you can Implement event  dispatching to help overcome this Pitfall by   extending the application abstract class or using  another pre-built type such as the application   event publisher class that lets you publish events  right into spring context events can fire off to   another service that can exist inside your code  base or with minimal configuration externally   these are many use cases for event dispatching  it's vital that you know what they are the more   you know about it the better you can use event  dispatching to log events in your application   this can help with debugging and monitoring  depending on your environment you can come   up with exciting use cases such as sending logs  to a database for storage or a security service   event dispatching can notify users on other  systems when certain events occur this can   be used for sending email notifications when a  new user registers or SMS notifications when a   product is out of stock event dispatching can  coordinate the behavior of different parts of   your application for example you could use  event dispatching for adding a student to a   university and a roster for a university class  and two separate databases in your architecture   you'll likely start with this approach as it's  the simplest implement you'll rely on events to   execute other parts of your code and only verify  the event was triggered in a unit or integration   test event dispatching is a powerful technique  that can make your application more modular   reusable and easier to test if you still need  to start using event dispatching I encourage   you to try it you may be surprised how much it  can improve your code the caveat being is that   it can also complicate your code if it isn't  clear that other events are being triggered   now when you start leaning towards an event driven  architecture there are some questions you should   ask yourself how resilient are my events can  I scale my systems depending on the use case   how do we make events more resilient a great way  to make a more resilient event driven architecture   is to offload your events to another service  asynchronously this is ampq or Advanced messaging   queuing protocol at the heart of asynchronous  communication protocols ampq is a protocol that   standardizes how messages should be transported  between applications like the shipping lanes   of our ocean mpq specifies how messages are  formatted addressed transported and confirmed   ensuring reliable and predictable point-to-point  or point-to-multipoint Communications the protocol   is designed around these Core Concepts exchanges  cues bindings and topics exchanges are entry   points for messages there are akin to Ports of  countries or messages or ships first arrive from   a publisher cues are buffers that store messages  if exchanges are ports cues are the harbors where   our ships are stored until they can be processed  messages are processed in order first in first out   just like the grocery check outline bindings are a  type of role that exchanges use to Route messages   into queues think of them as InterContinental  routes that defined where each ship should go the   routing key determines which column the message  should go to whether it's one queue or many   ampq has different types of exchanges namely  four direct topic fan out and headers typically   you hear about topics a form of routing where  messages are routed based on a match between a   message routing key and the pattern used to  bind a queue to an exchange the routing key   is a short string the message producer generates  specifying the topic to which the message belongs   it can contain wow cards for example the routing  key system.status would match the pattern   system.astric wildcard which would route the  message to any queue bound to the exchange with   the same pattern direct exchanges also allow the  message to be routed on the specific routing key   it's a simple unicast routing if you have a queue  whose binding is the same that's where it will go   fan out exchanges where the routing key is ignored  the message is sent out to all cues bound to The   Exchange there is a scenario where you need to  send a message to multiple cues this is the one   for you header exchanges rely on header values  instead of routing keys of course for all these   messages being routed you'll have Publishers and  consumers of those messages Publishers create   messages and send them to exchanges exchanges  use bindings are rules to decide how to route   messages to cues that end up at a consumer a  monitoring service can subscribe to a topic   that starts with a key and a wild card and consume  relevant messages such as system.status one of the  

major strengths of ampq is its reliability the  protocol guarantees messages delivery through   acknowledgment if a message isn't acknowledged  the protocol knows the message wasn't correctly   delivered and tries again this ensures that no  message gets lost and to stay analogous is to   say it ensures our ships aren't lost and the vast  ocean void enhancing the communication and system   reliability in an event-driven architecture the  reliability of messages is essential ensuring   an event was received as an integral part of  designing an ADA now let's talk about RAB and   mq a popular message broker that implements the  ampq protocol rapid mq at its core is an open   source message broker software that implements the  advanced message queuing protocol it's designed to   enable Communications between different parts of  a system if you wanted to ensure your spring blue   application events were robust and communicate  between dispersed and separate systems you'd   use a message broker like rabbitmq if you have  a server located somewhere else or want to share   with a system in the cloud such as an S3 bucket or  a database in a kubernetes pod picture rabbitmq as   an Intergalactic post office your applications or  planets if we stick to our Cosmic metaphor sends   messages to the message broker rabbitmq ensuring  they are delivered to the correct recipients no   matter where their system is located in the solar  system one of the greatest strengths of rabbitmq   is its support for various messaging patterns  rabbitmq has work or task queues if you have a   resource intensive task you don't want to wait to  complete you can send it as a message to a work   queue where multiple workers will execute it and  it does all the functions you would expect from   ampq message broker such as publish subscribing  routing and using topics rabbitmq is resilient   to failure it guarantees that a message will  be delivered even if the receiver is not ready   when it's sent or if some other parts of the  system have crashed rapidmq can be integrated   with spring boot enabling your applications  to communicate using events resiliently   spring boot provides abstractions and Integrations  to work with rampa mq namely the spring and PQ   project it provides templates for sending and  receiving messages containers and makes it   easier to implement in your project rabbitmq added  stream support in version 3.11 they offer large   fan outs allowing multiple applications to read  the same message it also has time travel so that   streams can replay data streams have timestamps  where a part of the stream can be picked and the   data replayed streams are persistent and in  rabbitmq replicate a non-destructive append   Only log these features for rapidmq are a game  changer and helped it to compete with Apache Kafka   normally in a rabbitmq scenario messages are  deleted from the queue whenever a message is   consumed however streams allow a consumer to  go to any point in a log and read from there   rabbitmq was designed to have any empty State  however now with stream support you can have large   backlogs that store data there's so much more to  rabbitmq it's a great option for anyone looking to   implement a message broker into their architecture  now let's take a deeper look into Apache Kafka   Apache Kafka is a distributed event streaming  platform born out of LinkedIn to handle their   growing data and activity and simple terms Kafka  allows us to send messages from one application   to another from one system to another in real  time using a publish And subscribe model as well   much like a courier service it ensures our  messages or events reach their intended recipient   but Kafka is not just a messaging system it's  a streaming platform that can store process   and analyze streams of Records in real time it  can handle trillions of daily events making it   a favor among Tech giants like Twitter Netflix  and Uber Apache Kafka is particularly valuable   and modern applications for several reasons its  ability to handle real-time data makes it ideal   for event-driven architecture where the goal is  to detect and react to changes in the state of   data as soon as they occur let's say you have a  retail application every time a customer places   an order this generates an event Kafka and can  ingest this event and notify various parts of   the application perhaps the event system needs to  update its stock the Billing System needs to issue   an invoice and the customer needs a confirmation  email this can be handled with Kafka allowing for   real-time efficient processing of course the  billing and inventory and customer systems   would listen for the event when each received the  order event they would act whether it was updating   their stock issuing an invoice or sending an  email confirmation the retail application is   effectively decoupled from the rest of the  system we could scale and evolve each system   independently of each other you can then link  this information to a visualization dashboard   to view real-time analytics and provide  your organization with business insights   events would be stored in a queue your dashboard  of choice or multiple dashboards would subscribe   to the queue and all this would happen in real  time Kafka allows for replaying events as it   maintains a log of all the events that have passed  to it this can be useful for debugging recovery   during system failures or different use cases such  as machine learning that may need access to its   data in different ways how long you can keep your  data can be configured using a retention policy   integrating Kafka with your application can take  it to the next level a distributed streaming   platform like Apache Kafka can create a robust  and resilient architecture for real-time data   processing streaming can perform machine learning  real-time anomaly detection and data enrichment   data enrichment is simply taking data from one  system and then adding data to it from another   source suppose you have a database that stores  user profiles and another that stores customer   order data this data is combined into a single  data structure and returned to a consumer such   as the user's current cart the shopping cart  service would subscribe to the user profile   and transaction topics this data could then  be enriched inside the service to display a   profile and Order data a combination of decoupled  systems coming together you can update Legacy   systems with minimal Interruption to users  transform and filter data in real time and   connect systems through a publish And subscribe  model a good thing to know about Apache Kafka   is that it uses connectors to connect to external  systems Source connectors can extract data from a   database send it to Apache Kafka and then a sync  connector can take that data from Apache Kafka   and put it into another database syncs usually  act on the data you are working with in some way   you can track changes to the database using change  data capture CDC which comes in two flavors either   you can use CDC logs or CDC query CDC logs examine  the database transaction or bin logs the types of   logs will vary depending on the database logs are  performant and aware of deletes in the database   however it can require special permissions to set  up you can also use CDC query it's less performant   and relies on sending a query because of this  it can only know what is inside of the database   earlier I mentioned spring Cloud dataflow  spring cloud dataflow is a powerful Cloud   native toolkit that allows for developing and  managing data integration and real-time data   processing and pipelines if you're building  a spring boot application it's a no-brainer   organizations deal with massive volumes of data  and the ability to efficiently process transform   and analyze data is critical now spring Cloud  dataflow provides a comprehensive solution to   these challenges offering a flexible and scalable  platform for building and managing data Pipelines   it has security by standardizing and oau-2 and  openid connect or authentication and authorization   it enables continuous delivery and allows  for upgrades and rollbacks of your Pipelines   now data flow serves as a great way  to enable an event-driven architecture   it's also used with spring Cloud Stream app  starters and has 60 pre-built sources so   developers can kick-start their Solutions that's  JMS Source mongodb rabbitmq gemfire cloud dataflow   is also an open source framework built on top  of the popular spring Boot and spring Cloud   projects it provides a platform for data-driven  microservice applications you can compose and   orchestrate data processing pipelines known  as streams and tasks based on applications   to handle batch processing it supports multiple  programming models such as Cloud stream and task   the platform is also polygon supporting Kafka  python.net and other technology polygot simply   means that the platform can support multiple  languages dataflow offers a plugable message   broker including support for rapidmq Apache Kafka  and Amazon kinesis it also has monitoring and   management features to gain insights into  the pipeline's performance monitor metrics   troubleshoot issues and dynamically adjust the  pipeline to optimize data processing it's a great   tool for orchestrating streams you can have  it batched in workflows scale data pipelines   and orchestrate tasks combined with other tools  and Technologies you can create an event-driven   architecture with a rabbitmq or Apache Kafka and  a spring cloud dataflow developing and learning   on event driven architecture will always level  up your skills I would be remiss if I didn't   mention how you as a software engineer can get  these Technologies in the cloud whether it's AWS   Google cloud or Azure AWS offers Amazon mq Apache  activemq rabbitmq Amazon Kinesis Google cloud has   uh Cloud Pub sub Cloud dataflow Azure has Azure  service bus event Hub and event grid as your skill   set grows as software engineer you need to provide  solutions to your company and the best solution   may be to host your services and kubernetes or  to move them to a managed service let me know   what service you decided to go with and how you're  implementing your event-driven architecture this   has been another episode of the Joshua Matos show  thanks for sticking around and I'll see you later

2023-07-28

Show video