Microservices Observability, Resilience, Monitoring on .Net

Docker Commands

docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d

docker-compose -f docker-compose.yml -f docker-compose.override.yml up --build

docker-compose -f docker-compose.yml -f docker-compose.override.yml down

You can launch microservices as below urls:

• Catalog API -> http://host.docker.internal:8000/swagger/index.html

• Basket API -> http://host.docker.internal:8001/swagger/index.html

• Discount API -> http://host.docker.internal:8002/swagger/index.html

• Ordering API -> http://host.docker.internal:8004/swagger/index.html

• Shopping.Aggregator -> http://host.docker.internal:8005/swagger/index.html

• API Gateway -> http://host.docker.internal:8010/Catalog

• Rabbit Management Dashboard -> http://host.docker.internal:15672 -- guest/guest

• Portainer -> http://host.docker.internal:9000 -- admin/admin1234

• pgAdmin PostgreSQL -> http://host.docker.internal:5050 -- [email protected]/admin1234

• Elasticsearch -> http://host.docker.internal:9200 -- To Be Develop

• Kibana -> http://host.docker.internal:5601 -- To Be Develop

• Web Status -> http://host.docker.internal:8007 -- To Be Develop

• Web UI -> http://host.docker.internal:8006

5. Launch http://host.docker.internal:8007 in your browser to view the Web Status. Make sure that every microservices are healthy.

6. Launch http://host.docker.internal:8006 in your browser to view the Web UI. You can use Web project in order to call microservices over API Gateway. When you checkout the basket you can follow queue record on RabbitMQ dashboard.

Introduction - Distributed Logging with Elastic Stack (Elasticsearh + Logstach + Kibana and SeriLog)

Start Docker compose :

docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d

docker-compose -f docker-compose.yml -f docker-compose.override.yml down

Go to -> AspnetRunBasics

Install Packages

Install-Package Serilog.AspNetCore

Install-Package Serilog.Enrichers.Environment

Install-Package Serilog.Sinks.Elasticsearch

Check item group project file

  <ItemGroup>

    <PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.10.9" />

    <PackageReference Include="Serilog.AspNetCore" Version="3.4.0" />

    <PackageReference Include="Serilog.Enrichers.Environment" Version="2.1.3" />

    <PackageReference Include="Serilog.Sinks.Elasticsearch" Version="8.4.1" />

  </ItemGroup>

Configure SeriLog

After that, we will configure logging in Program.cs by adding the following details on Main method.

AspnetRunBasics

Program.cs -- UseSerilog --- ADDED

public static IHostBuilder CreateHostBuilder(string[] args) =>

           Host.CreateDefaultBuilder(args)     

              .UseSerilog((context, configuration) =>

               {

                   configuration

                    .Enrich.FromLogContext()

                    .Enrich.WithMachineName()

                    .WriteTo.Console()

                    .WriteTo.Elasticsearch(

                        new ElasticsearchSinkOptions(new Uri(context.Configuration["ElasticConfiguration:Uri"]))

                        {

                            IndexFormat = $"applogs-{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{context.HostingEnvironment.EnvironmentName?.ToLower().Replace(".", "-")}-logs-{DateTime.UtcNow:yyyy-MM}",

                            AutoRegisterTemplate = true,

                            NumberOfShards = 2,

                            NumberOfReplicas = 1

                        })

                    .Enrich.WithProperty("Environment", context.HostingEnvironment.EnvironmentName)

                    .ReadFrom.Configuration(context.Configuration);

               })

               .ConfigureWebHostDefaults(webBuilder =>

               {

                   webBuilder.UseStartup<Startup>();

               });

-- We have configured Serilog according to writing Console and Elastic Search. When Configure for the Elastic Search we provide some sink options.

Test SeriLog For ElasticSearch and Kibana Sink Integration in Shopping Web Microservices

Create SeriLog Common Logging Library For ElasticSearch and Kibana Sink Integration to All Microservices

Adding SeriLog Common Logging Library Project References to Aspnetrun Shopping Web Application

Adding LoggingDelegatingHandler for Intercepting Microservices Request-Response Logging on ElasticSearch and Kibana

Adding SeriLog for Shopping.Aggregator Microservices for Logging on ElasticSearch and Kibana

Adding SeriLog for Catalog.API Microservices for Logging on ElasticSearch and Kibana

Adding SeriLog for Basket.API Microservices for Logging on ElasticSearch and Kibana

Adding SeriLog for Discount.API Microservices for Logging on ElasticSearch and Kibana

Adding SeriLog for Discount.Grpc Microservices for Logging on ElasticSearch and Kibana

Adding SeriLog for Ordering.API Microservices for Logging on ElasticSearch and Kibana

Adding SeriLog for OcelotApiGw Microservices for Logging on ElasticSearch and Kibana

Containerize All Microservices with SeriLog using Docker Compose for Logging on ElasticSearch and Kibana

Containerize All Microservices with SeriLog using Docker Compose for Logging on ElasticSearch and Kibana Part 2

Test on Docker environment - SeriLog Microservices into Docker Compose for Logging on ElasticSearch and Kibana

Apply Retry Pattern with Polly policies on HttpClientFactory for Shopping.Aggregator Microservices

Apply Circuit Breaker Pattern with Polly policies on HttpClientFactory for Shopping.Aggregator Microservices

Develop Advance Policies for Retry and Circuit Breaker Pattern with Polly policies on HttpClientFactory

Nuget Package

Install-Package Microsoft.Extensions.Http.Polly

See that

<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="5.0.1" />

Apply Policies for Retry and Circuit Breaker Pattern with Polly policies for AspnetRun Shopping Microservices

Nuget Package

Install-Package Microsoft.Extensions.Http.Polly

See that

<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="5.0.1" />

Using Polly for Database Migration Retries for Ordering.API EF.Core Sql Server Migration

Using Polly for Database Migration Retries for Discount.API/Grpc Dapper PostgreSQL Migration

Nuget Package

Install-Package Microsoft.AspNetCore.Diagnostics.HealthChecks

Adding json response

Nuget Package

Install-Package AspNetCore.HealthChecks.UI.Client

Startup.cs -> Configure

app.UseEndpoints(endpoints =>

            {

                endpoints.MapControllers();               

                endpoints.MapHealthChecks("/hc", new HealthCheckOptions()

                {

                    Predicate = _ => true,

                    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse

                });

            });

Adding Health Check for Basket.API Microservices with Checking Redis and RabbitMQ Connection

Adding Health Check for Ordering.API Microservices with Checking Sql Server and RabbitMQ Connection

Adding Health Check for Shopping.Aggregator Microservices with Checking Internal Microservices Integrations

Adding Health Check for AspnetRunBasics Shopping Microservices with Checking Ocelot ApiGw Integrations

Developing WebStatus App for Centralized Microservices Health Monitoring Using Watchdogs for Visualization

Developing WebStatus App for Centralized Microservices Health Monitoring Using Watchdogs for Visualization Part 2

appsettings.json file

"HealthChecks-UI": {

    "HealthChecks": [

      {

        "Name": "Catalog Health Check",

        "Uri": "http://localhost:8000/hc"

      },

      {

        "Name": "Basket Health Check",

        "Uri": "http://localhost:8001/hc"

      },

      {

        "Name": "Discount Health Check",

        "Uri": "http://localhost:8002/hc"

      },

      {

        "Name": "Ordering Health Check",

        "Uri": "http://localhost:8004/hc"

      },

      {

        "Name": "Shopping Aggregator Health Check",

        "Uri": "http://localhost:8005/hc"

      },

      {

        "Name": "AspnetRunBasics WebMVC Health Check",

        "Uri": "http://localhost:8006/hc"

      }

    ],

Test All Microservices on Docker environment - WebStatus Health Monitoring Microservices into Docker Compose for Visulize WatchDog HC

Thanks Mohammed Mansoor for developing this assignment. He developed the distributed tracing with OpenTelemetry but I used Jaeger as a tracing tool.

See Codes on Github -> https://github.com/mansoorafzal/AspnetMicroservices

See Thread in Q&A -> https://www.udemy.com/course/microservices-observability-resilience-monitoring-on-net/learn/#questions/15889908/