AWS Serverless Design for IoT

An overview of the serverless design model for IoT data on AWS that we will focus on in this course.  Serverless services used are centered around API Gateway, S3, IoT Core, DynamoDB, and Lambda.

A brief discussion of my other AWS IoT course on Udemy and how it varies from this course.  This will be a useful lecture to determine which course is most appropriate and aligned with the students interests.

A discussion of prerequisites for the course and of the technical background you should posses before purchasing and embarking on the course.

A brief discussion of serverless "frameworks" available and there utility.  We will not be using serverless frameworks in this course buy they are important to know about if youintend to pursue serverless computing on the cloud.

A quick discussion of what IoT project and real world use cases you may experience as an IoT engineer.  In this course we cover simple use cases, but these use cases can become complex, covering various devices, prodigious code, and extensive cloud services.

A lecture going over the basic communication protocols and security considerations enabling devices to talk to AWS.  We specifically discuss MQTT, HTTPS, SigV4, and SS/TLS requirements for device to cloud communication.

In this video I discuss cosmetic updates to AWS IoT Core for 2023. The main update is the "Actions" blade has been changed to a blade called "Message routing" and IoT "Rules" can now be created directly from this blade without the pretty pictures for each AWS service that appeared previously before these cosmetic changes to AWS IoT Core took place in 2023.

In this video I show two different methods for sending fake test data in JSON format to AWS IoT Core.  These methods are useful if either you don't have your device yet or you don't want to set up your device just to send some quick data for testing purposes.

In this lecture we describe the baic sketch which we will base our ESP8266 and ESP32 device implementation on to talk to AWS IoT.  There is a link to the original sketches which allow a single page application with embedded certificates to connect to AWS IoT Core and transfer data.  Also we will discuss installation directions for the boards manager and firmware packages as well as the PubSub library for MQTT.

In this lecture we continue to work on our ESP8266/ESP32 single page sketch to communicate with AWS IoT Core.  We discuss how to configure our personal AWS endpoint as well as replace our devices private certificate, and client certificate with our own device certificates downloaded from AWS IoT Core (See appendix/YouTube video link where I explain this if you don't already know how to generate device certificates).   We can use our own x.509 CA root certificate or use the one provided by the sketch.  We also discuss making things easier by using the EOF technique to avoid meticulous certificate formatting issues.

Now that we have configured the Arduino sketch for our account and endpoint with our device certificates we can now connect our device an speak to AWS IoT Core directly from our ESP8266/ESP32 device.

In this lecture we discuss how to publish from the cloud to a subscription topic on the device.  This built in benefit of MQTT allows inherent duplex communication but using a topic to publish from the device but also another topic to publish from the cloud.  This feature is useful if we ever need to change device state based on cloud events.  Also I demonstrate a method to generate fake data from a virtual device on AWS IoT Core if you don't have a physical device available.

An Introduction to AWS IoT Analytics which is a managed AWS serverless solution using underlying AWS services including AWS IoT Core, Kinesis, DynamoDB, and S3.

Creating a AWS IoT Analytics data-flow configuring our Channel, Pipeline, and Datastore.

We discuss altering our hello world sketch from the last lecture to now send a data packet to AWS IoT Core that provides temperature and humidity via a snprintf data buffer.

A deeper discussion into AWS Lambda and how it functions when dealing with IoT data.  Specifically how we can use Lambda data enhancement in the AWS IoT Analytics pipeline to process, filter, compute, or enhance our IoT data.  An important overview of lambda in general is discussed as it is the key component of the AWS serverless model.

In this lecture we test a Lambda function in Node.js with an incoming payload from the test function within lambda.  We go over explicitly extracting both the incoming 'event' object as well as the 'context' object from an incoming data payload.   

Inthis lecture we note the difference between a batch of data coming in from the pipeline in AWS IoT Analytics and what will come in directly from AWS IoT Core later in the course.  The batched JSON from the underlying AWS Kinesis service has an additional nesting in the JSON structure.  We also show how to set our Lambda permissions through the AWS CLI.  Finally we run our simPub1 Arduino script on our device to produce out data set for using in AWS QuickSight and then AWS SageMaker.

In this lecture we use AWS QuickSight with the built in connector to our AWS IoT Analytics service to ingest the data for a customized visualization.  We use a simple line chart but many visualizations for BI are available for you to manipulate the IoT date from your Data Set in IoT Analytics.

Now that we produced our data set from our device sending IoT data and having that data enhanced with Lambda we had a chance to use AWS QuickSight for some visualizations.  In this lecture we can use a simple Python program to provide addition visualizations with minor statistical analysis on our devices IoT data by invoking the Data Set we created previously and utilizing the Pandas and Matplotlib librarys

In this lecture we discuss the pros and ons of both utilzing an invoking our Data Set directly from IoT Analytics as opposed to creating a data lake repository of our data held in a AWS S3 bucket.  Many more AWS services can work on a S3 bucket as well as additional advantages discussed in this lecture.

In this lecture we now duplicate our IoT data, not only in our Data Store and Data set for consumption by other AWS services, but we also send our IoT data in CSV format to a bucket repository in S3 which will function as a data lake.  Our data in S3 is now set to be consumed by any outside or internal service we like as long as bucket permissions are set.

In this video I show you how to create a completely open and accessible bucket on AWS S3 for 2022  with a static IP.   Once we create our accessible public bucket we can take one extra step to make it serve as a static web host as well as a IoT data bucket. We will set the Bucket Policy to 'read only' and then design an accessible CORS rule to allow cross-origin access with the appropriate headers.

In this lecture we set up our S3 bucket which holds our IoT data to be public to anyone who wants to read it.  This is fine as long as our data is not confidential as we are only given read permission not delete or write permission. 

We demonstrate a Google Chart hosted on a remote testing environment to ensure it can consumes our IoT data from another S3 bucket and can graph the IoT data  given the serverside timestamp on the x-axis.

We demonstrate a Google Chart hosted in a public S3 bucket which consumes our IoT data from another bucket holding our IoT data and graphs it over the serverside timestamp on the x-axis enhanced in our lambda function. The temperature and humidity are charted on the Y-Axis.

An introduction to our advanced example on AWS IoT Analytics.  Here I discuss the design flow en toto, and how we will be using various AWS services to transmit, enhance, store, retrieve, and visualize out IoT Data.

In this lecture I discuss the first of our two Lambda functions written in Node.js.  This Lambda function will enhance and decode our incoming GPS data to give use the current weather and city/state for our incoming GPS coordinates.  To accomplish this we will be 'rolling up' the Weather-JS and Cities Node modules with NPM to provide the needed functionality 

In this lecture we will first test our Lambda function from Lambda itself, and then test it in from within  the AWS IoT Analytics design flow.  Once in the design flow, we will give AWS IoT Analytics permission to execute our Lambda by using the AWS CLI to grant permission.

In this lecture we go over our Arduino Sketch, and show how to create a data package within our Arduino sketch which transmits both internal pseudo-weather and pseudo-GPS data to AWS IoT Core via MQTT.  Once our IoT data is in IoT Core, it will be connected to AWS IoT Analytics via the 'Channel' in IoT Analytics that we created previously

We focus in further on our design flow for the last few parts of our IoT serverless architecture in which we create our second lambda, API Gateway endpoint, and static website hosted on S3.

In this lecture we design our second lambda function which allows us to extract IoT data from our private data bucket held in S3.  This lambda will first extract the CSV data using the AWS-SDK with a S3 object, and then  transform it to JSON format for better consumption on our static website

In this lecture we create a URL endpoint that we link to our extraction Lambda.  Once our GET request is invoked from our API Gateway created endpoint, we can then ingest the transformed data from our Lambda function to our browser, Postman, or webhost.

In this lecture we design our static web page in Html/JavaScript and invoke our URL endpoint previously created with API Gateway.  Once the URL is invoked we can retrieve our data via lambda and then place it in our Highcahrts visualization.  We will visualize both the internal and external temperature and humidity,  as well as generate maps with google given our coordinates sent  from our device and stored in our data lake on S3.

A quick preview for adding security by adding an API Key.  This material is covered in depth toward the end of the course, however you can also get the secured website for preview here.

In this video we create an Action and Rule in AWS IoT Core which sends data to a S3 bucket we are using as a static web host. No storage is needed for this simple design.  This is a Highcharts visualization hosted on S3 that works with an IoT Core action that saves data to S3. The JavaScript code in the S3 webhost fetches the IoT data in the S3 bucket and synchronously charts the resultant data points for each fetch interval. The cost should be a few pennies if you are off the free tier.

In this video we upload our HTML and JavaScript code to our webhost in S3.  We then test the code from the MQTT test client in AWS IoT Core to insure it is producing our Highchart visualizations.

In this video we use our new AWS IoT Core Action/Rule in AWS IoT Core which sends data to our S3 bucket. We sent test data with a customized bash script provided by AWS to automatically produce data with editable iterations and delays between IoT JSON payloads.

A discussion of different levels of data ingestion and consumption techniques.  After going over simple "level" one design utilizing lambda ingestion from AWS IoT Core.  We can also implement IoT Data ingestion that originates from API Gateway.

Brief overview of serverless methods for IoT Data.  In this section we focus on a simple IoT data dispatch to s3 utilizing a Lambda function written in Node.js.  Specifically using Lambda to dispatch our incoming event object from AWS IoT Core to S3 for a data Lake as an IoT data repository.

Writing a Node.js function in Lambda to choose a bucket and file for our IoT data object.  We will have one data object per folder as a first example.

Testing our Lambda function and assigning appropriate IAM permissions to write from Lambda to S3.  We send a fake data package from AWS IoT Core but attached is the script for both your ESP8266 and ESP32 to send the data package I demonstrate from AWS IoT Core on the MQTT broker.   Note the topic name in the sketch we publish from we will need to subscribe to on the device.

Now we change or Lambda function to create multiple timestamped events and place them into the same we created earlier.

Making our Static website public and setting our permissions, bucket policy, and CORS.  Then using three files (index.html, main.js, and visualizer.js)  with Chart.js we can now  visualize our IoT data using the format generated by our modified Lambda function in the third lecture of this section.  If you like you can use your own webhost and do your own website based ouround your own IoT data.

Introduction to Kinesis Data Firehose and using this service with IoT

We cover how to configure Kinesis Data Firehose for IoT data sent from our device, and then we look at the specific schema that is saved to our S3 data lake as a repository for our Kinesis data.  Notice the JSON is saved in batched format similar to IoT Analytics when it was enhanced in Lambda.  This is because Kinesis Firehose was the underlying service in the data pipeline.

Now that we have figured out what our data schema looks like on the S3 repository we can design a Chart.js program that gives us a Bar chart and line graph for basic visualization.

We introduce our project for level two IoT Serverless design.  We base our project on the Guitar Room environment sensor project.  This project is based around a practical implementation of the AWS JavaScript SDK in the browser.  We add Lambda enhancement as well as utilize DynamoDB, IoT Core, Cognito, and a S3 static Web Host.

In this lecture we format our database in DynamoDB to have the Schema reflect the data that will be passed on from Lambda. 

We develop a Lambda function to add a properly formatted ISO 8601 timestamp.  This function takes the environmental JSON event object from AWS IoT Core and modifys it to pass to our DynamoDB table in the proper format.  We need to insure our data is formatted correctly by testing a JSON test package directly from the Lambda function.

In this lecture we connect our Lambda fuction to AWS IoT cCore to accept our  enviro data from our ESP8266/ESP32 device.  This new action from IoT Core will send the JSON payload as an event to be enhanced by our Lambda function from the previous lecture.

In this lecture we create a simple AWS Cognito  Identity in JavaScript which we can insert directly into our static web host.   This now provides permissions which will allow our static web host to read our IoT data lake from S3.   This unauthorized Cognito Identity will then be provisioned in IAM with an attached S3 read-only policy.

In this lecture we upload our index.html and refresh.js as static web pages held in a public S3 bucket.  Once these pages are uploaded we will further examine our refresh.js page and discuss how the DynamoDB query is handled by using the AWS JavaScript SDK invoked directly from the browser.  We will also discuss refresh timing, as well as macthing our variable names with our DynamoDB table and Lambda function.

We introduce level three IoT Serverless design utilizing Lambda to invoke AWS services to retrieve IoT data from our data repositories.  I explain the pro's and con's of using this method over our previous level two and level three methods to design a serverless IoT process on AWS.

Here we set up out Lambda function to retrieve our IoT JSON data objects from our data repository held in S3.

Using API Gateway with Lambda integration we create an accessible URL endpoint, tied to our Lambda function, that we can use in our static web host.

Here we create a static webhost in JavaScript and HTML and utilize our API Gateway endpoint to invoke our Lambda function and retrieve our IoT data from S3 so that we can visualize the data with Highcharts

In this lecture we clean our IoT data and in turn demonstrate how we can add GetObject and ListObject permissions in our Lambda role so that we can now use a private bucket to host our IoT data in S3 to demonstrate the ability to keep our data private as needed.

In this Lambda example we invoke the AWS-SDK in Node.js to ingest an incoming IoT data payload from IoT Core, and then republish that JSON data to another topic on a different IoT device. The receiving device simply needs to subscribe to a matching MQTT topic sent from lambda to receive the data payload on the device.

In this lecture we first describe the need for security on AWS and then demonstrate a quick and easy method of IP address limiting.  Using this technique we will compose a new Bucket Policy to whitelist specific IP(s), giving those explicitly listed IP addresses the ability to view our visualization web sight hosted on s3.  This method will effectively obfuscate an unauthorized viewer, from non-allowed IP address, from acquiring our API endpoint, and thus preventing access to our Lambda function and IoT data repository.

In this lecture I show you how to create an API Key and Usage Plan in AWS API Gateway.  Once our plan and API Key are created we can assign it to an API Gateway deployment in which our endpoint will require the API key in the GET method in the header (setRequestHeader) response from our web sight.  If we enter an incorrect API key it will return an "access denied" .  If we supply the correct key then the endpoint will allow API Gateway to execute our Lambda and extract data from our s3 bucket for our level 3 design.  We will test our newly created key with a GET request in Postman

In this lecture we examine our visualization web sight HTML/JS code to examine how to insert the proper DOM object and make a function to use the setRequestHeader with our API Key using text box entry and event listener in which the API Key is supplied to a trusted client to use our API Gateway provided endpoint.

Introduction to AWS Asynchronous IoT, and how AWS WebSockets works with Lambda to provide true Real-Time performance for our simple visualization. This project is similar to project one as far as the S3 web host visualization is concerned. However, to make this asynchronous, I introduce AWS WebSockets along with Lambda and API Gateway to use a “server push” rather than a “client pull” model. IoT data is pushed via a WebSockets endpoint in lambda through API Gateway. We also need to refactor our JavaScript code to make use of the Websockets API with our mapped 'connection' and 'sendIoTpayload' lambda functions.

In this video we create our "connection" and "sendIoTpayload"  lambda functions , give the later  "APIExecute permissions." After this we create our WebSocket mappings in API Gatewayand then  finally  integrate our "sendIoTPayload"  Lambda with our WebSocket endpoint

In this lecture we further modify our 'connect' and 'sendIoTPayload' lambdas and send test messages to some minimal Web code to test our WebSocket connection.

We finalize our webcode for a full visualization and test it with the MQTT test client to publish sample data.  After this we can use our Bash test script to emulate a device and send multiple randomized JSON IoT payloads to the web host.

In this lecture we discuss improving and automating our lambdas using the AWS Systems Manager Parameter Store along with AWS WebSockets. We hope to remove the need for the less elegant manual saving of the connection_identification and see that we can automate the IoT design flow. This project is similar to project one and two as far as the S3 web host visualization is concerned we just need to utilize a couple extra API's with the Systems Manager.

In this lecture we go over both the 'connectionimproved.js' lambda function  and the 'sendiotpayload2.py'  lambda function and discuss the changes to SSM Get and Put API's using the AWS Systems Manager Parameter Store for inter-Lambda communications.

Remember our 'connectionImproved" lambda only needs "Systems Manager" permission while our "sendIoTpayload2" function needs both "Systems Manager" permissions and "ExecuteAPI" permissions so the post_to_connection() API works.

In this video We set up our familiar action in IoT Core to AWS Lambda and then use our test bash script to send readings to visualize our data readings on the line chart.  We can use the S3 web host we made earlier, but instead here I use an external website as a temporary  web host for our code in order to demonstrate that we only need the 'wss://websocketaddress.com' to connect to our entire tool chain in AWS with the external website.

Designing a lambda function in Node.js and testing it to insure it is sending data to a s3 bucket.

Setting up API Gateway with Proxy integration to link to our Lamda function from the last lecture.  The API Gateway proxy will send the entire Data package from our device sketch in Arduino to Lambda to S3.

Here we use a modify Arduino Sketch to connect your ESP device to API Gateway directly using SHA thumbprint security.  Normally the API Gateway endpoint is accessed via HTTPS on a PC which natively provides security.

Demonstrating charting  and visualization for our IoT data held in S3 using both Chart.js and Highcharts.js.  Don't forget to alter the files to point to the target bucket and folder/keys to where your IoT data resides.

In this lecture we create an action in IoT Core to send our incoming JSON IoT data payload to AWS Timestream. First we must create our database and then our data table in Timestream, which we can do directly from IoT Core. Now that Timestream is set up and able to accept data we input or data through IoT Core and then query the data in Timestream using basic SQL statements.

Now that our Timestream data ledger is full of our IoT data we can query it and create visualizations with Grafana.

First we make a free Grafana online account, then we download the Timestream 'connector' designed to allow Grafana to read from Timestream in our account. We will create some limited credentials (SigV4) in AWS IAM to give Grafana the privilege's needed to read our Timestream data in AWS. Finally, now that we have formed the connection between Grafana and Timestream, we can do a few visualizations using the built in query capability of Grafana on our Timestream data.

Background info on signing up for a free tier AWS account and installing the AWS Command line tool called the AWS CLI, which requires your Account ID and Secret key.  aws1 or aws2 is fine.

In this lecture we discuss the AWS IoT Core service, using the 'monitor' and 'test' consoles, and linking actions from AWS IoT Core to other important AWS IoT related services.

A short introduction to the often confusing concept of IAM policy's and roles. Here we create a super expansive IoT policy with the automatic configuration tool in AWS IAM as a test case before creating an real IoT policy in AWS IoT core the simple way.

In this lecture we use the 'one-click' policy creation tool for AWS IoT core, then configure our three required security certificates for use in our devices. After we download and activate our credentials we attach the inclusive IoT policy we just created to those credentials.