Integrating Datadog with AWS and Installing the Datadog Agent on EKS

We integrated Datadog with AWS and deployed the Datadog Agent on our EKS cluster. This allowed us to collect logs, metrics, and traces from both cloud infrastructure and container workloads.

Prerequisites

• We used an AWS account with admin permissions.
• We signed up for a Datadog account (free trial).
• We had an existing Amazon EKS cluster.
• We ensured IAM permissions were in place for Datadog to access AWS services.

Part 1: Integrating Datadog with AWS

Step 1: Set Up the Datadog AWS Integration

1. Navigate to Datadog AWS Integration
   • We logged into our Datadog account (datadoghq.com)
   • We navigated to Integrations → AWS.
2. Configure AWS Integration
   • We clicked Add Account, entered our AWS Account ID,
   • We selected the AWS services (EC2, S3, Lambda, etc.) to monitor.
   • We selected the metrics you want to collect (EC2, S3, Lambda, etc.).
3. Set Up IAM Role for Datadog

   • In AWS, We created an IAM role using Datadog AWS policy (or use the auto-generated CloudFormation template from Datadog).

   • We attached the following permissions:

     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Action": [
                     "apigateway:GET",
                     "aoss:BatchGetCollection",
                     "aoss:ListCollections",
                     "autoscaling:Describe*",
                     "backup:List*",
                     "bcm-data-exports:GetExport",
                     "bcm-data-exports:ListExports",
                     "bedrock:GetAgent",
                     "bedrock:GetAgentAlias",
                     "bedrock:GetFlow",
                     "bedrock:GetFlowAlias",
                     "bedrock:GetGuardrail",
                     "bedrock:GetImportedModel",
                     "bedrock:GetInferenceProfile",
                     "bedrock:GetMarketplaceModelEndpoint",
                     "bedrock:ListAgentAliases",
                     "bedrock:ListAgents",
                     "bedrock:ListFlowAliases",
                     "bedrock:ListFlows",
                     "bedrock:ListGuardrails",
                     "bedrock:ListImportedModels",
                     "bedrock:ListInferenceProfiles",
                     "bedrock:ListMarketplaceModelEndpoints",
                     "bedrock:ListPromptRouters",
                     "bedrock:ListProvisionedModelThroughputs",
                     "budgets:ViewBudget",
                     "cassandra:Select",
                     "cloudfront:GetDistributionConfig",
                     "cloudfront:ListDistributions",
                     "cloudtrail:DescribeTrails",
                     "cloudtrail:GetTrailStatus",
                     "cloudtrail:LookupEvents",
                     "cloudwatch:Describe*",
                     "cloudwatch:Get*",
                     "cloudwatch:List*",
                     "codeartifact:DescribeDomain",
                     "codeartifact:DescribePackageGroup",
                     "codeartifact:DescribeRepository",
                     "codeartifact:ListDomains",
                     "codeartifact:ListPackageGroups",
                     "codeartifact:ListPackages",
                     "codedeploy:BatchGet*",
                     "codedeploy:List*",
                     "codepipeline:ListWebhooks",
                     "cur:DescribeReportDefinitions",
                     "directconnect:Describe*",
                     "dynamodb:Describe*",
                     "dynamodb:List*",
                     "ec2:Describe*",
                     "ec2:GetAllowedImagesSettings",
                     "ec2:GetEbsDefaultKmsKeyId",
                     "ec2:GetInstanceMetadataDefaults",
                     "ec2:GetSerialConsoleAccessStatus",
                     "ec2:GetSnapshotBlockPublicAccessState",
                     "ec2:GetTransitGatewayPrefixListReferences",
                     "ec2:SearchTransitGatewayRoutes",
                     "ecs:Describe*",
                     "ecs:List*",
                     "elasticache:Describe*",
                     "elasticache:List*",
                     "elasticfilesystem:DescribeAccessPoints",
                     "elasticfilesystem:DescribeFileSystems",
                     "elasticfilesystem:DescribeTags",
                     "elasticloadbalancing:Describe*",
                     "elasticmapreduce:Describe*",
                     "elasticmapreduce:List*",
                     "emr-containers:ListManagedEndpoints",
                     "emr-containers:ListSecurityConfigurations",
                     "emr-containers:ListVirtualClusters",
                     "es:DescribeElasticsearchDomains",
                     "es:ListDomainNames",
                     "es:ListTags",
                     "events:CreateEventBus",
                     "fsx:DescribeFileSystems",
                     "fsx:ListTagsForResource",
                     "glacier:GetVaultNotifications",
                     "glue:ListRegistries",
                     "grafana:DescribeWorkspace",
                     "greengrass:GetComponent",
                     "greengrass:GetConnectivityInfo",
                     "greengrass:GetCoreDevice",
                     "greengrass:GetDeployment",
                     "health:DescribeAffectedEntities",
                     "health:DescribeEventDetails",
                     "health:DescribeEvents",
                     "kinesis:Describe*",
                     "kinesis:List*",
                     "lambda:GetPolicy",
                     "lambda:List*",
                     "lightsail:GetInstancePortStates",
                     "logs:DeleteSubscriptionFilter",
                     "logs:DescribeLogGroups",
                     "logs:DescribeLogStreams",
                     "logs:DescribeSubscriptionFilters",
                     "logs:FilterLogEvents",
                     "logs:PutSubscriptionFilter",
                     "logs:TestMetricFilter",
                     "macie2:GetAllowList",
                     "macie2:GetCustomDataIdentifier",
                     "macie2:ListAllowLists",
                     "macie2:ListCustomDataIdentifiers",
                     "macie2:ListMembers",
                     "macie2:GetMacieSession",
                     "managedblockchain:GetAccessor",
                     "managedblockchain:GetMember",
                     "managedblockchain:GetNetwork",
                     "managedblockchain:GetNode",
                     "managedblockchain:GetProposal",
                     "managedblockchain:ListAccessors",
                     "managedblockchain:ListInvitations",
                     "managedblockchain:ListMembers",
                     "managedblockchain:ListNodes",
                     "managedblockchain:ListProposals",
                     "memorydb:DescribeAcls",
                     "memorydb:DescribeMultiRegionClusters",
                     "memorydb:DescribeParameterGroups",
                     "memorydb:DescribeReservedNodes",
                     "memorydb:DescribeSnapshots",
                     "memorydb:DescribeSubnetGroups",
                     "memorydb:DescribeUsers",
                     "oam:ListAttachedLinks",
                     "oam:ListSinks",
                     "organizations:Describe*",
                     "organizations:List*",
                     "osis:GetPipeline",
                     "osis:GetPipelineBlueprint",
                     "osis:ListPipelineBlueprints",
                     "osis:ListPipelines",
                     "proton:GetComponent",
                     "proton:GetDeployment",
                     "proton:GetEnvironment",
                     "proton:GetEnvironmentAccountConnection",
                     "proton:GetEnvironmentTemplate",
                     "proton:GetEnvironmentTemplateVersion",
                     "proton:GetRepository",
                     "proton:GetService",
                     "proton:GetServiceInstance",
                     "proton:GetServiceTemplate",
                     "proton:GetServiceTemplateVersion",
                     "proton:ListComponents",
                     "proton:ListDeployments",
                     "proton:ListEnvironmentAccountConnections",
                     "proton:ListEnvironmentTemplateVersions",
                     "proton:ListEnvironmentTemplates",
                     "proton:ListEnvironments",
                     "proton:ListRepositories",
                     "proton:ListServiceInstances",
                     "proton:ListServiceTemplateVersions",
                     "proton:ListServiceTemplates",
                     "proton:ListServices",
                     "qldb:ListJournalKinesisStreamsForLedger",
                     "rds:Describe*",
                     "rds:List*",
                     "redshift:DescribeClusters",
                     "redshift:DescribeLoggingStatus",
                     "redshift-serverless:ListEndpointAccess",
                     "redshift-serverless:ListManagedWorkgroups",
                     "redshift-serverless:ListNamespaces",
                     "redshift-serverless:ListRecoveryPoints",
                     "redshift-serverless:ListSnapshots",
                     "route53:List*",
                     "s3:GetBucketLocation",
                     "s3:GetBucketLogging",
                     "s3:GetBucketNotification",
                     "s3:GetBucketTagging",
                     "s3:ListAccessGrants",
                     "s3:ListAllMyBuckets",
                     "s3:PutBucketNotification",
                     "s3express:GetBucketPolicy",
                     "s3express:GetEncryptionConfiguration",
                     "s3express:ListAllMyDirectoryBuckets",
                     "s3tables:GetTableBucketMaintenanceConfiguration",
                     "s3tables:ListTableBuckets",
                     "s3tables:ListTables",
                     "savingsplans:DescribeSavingsPlanRates",
                     "savingsplans:DescribeSavingsPlans",
                     "secretsmanager:GetResourcePolicy",
                     "ses:Get*",
                     "ses:ListAddonInstances",
                     "ses:ListAddonSubscriptions",
                     "ses:ListAddressLists",
                     "ses:ListArchives",
                     "ses:ListContactLists",
                     "ses:ListCustomVerificationEmailTemplates",
                     "ses:ListMultiRegionEndpoints",
                     "ses:ListIngressPoints",
                     "ses:ListRelays",
                     "ses:ListRuleSets",
                     "ses:ListTemplates",
                     "ses:ListTrafficPolicies",
                     "sns:GetSubscriptionAttributes",
                     "sns:List*",
                     "sns:Publish",
                     "sqs:ListQueues",
                     "states:DescribeStateMachine",
                     "states:ListStateMachines",
                     "support:DescribeTrustedAdvisor*",
                     "support:RefreshTrustedAdvisorCheck",
                     "tag:GetResources",
                     "tag:GetTagKeys",
                     "tag:GetTagValues",
                     "timestream:DescribeEndpoints",
                     "timestream:ListTables",
                     "waf-regional:GetRule",
                     "waf-regional:GetRuleGroup",
                     "waf-regional:ListRuleGroups",
                     "waf-regional:ListRules",
                     "waf:GetRule",
                     "waf:GetRuleGroup",
                     "waf:ListRuleGroups",
                     "waf:ListRules",
                     "wafv2:GetIPSet",
                     "wafv2:GetLoggingConfiguration",
                     "wafv2:GetRegexPatternSet",
                     "wafv2:GetRuleGroup",
                     "wafv2:ListLoggingConfigurations",
                     "workmail:DescribeOrganization",
                     "workmail:ListOrganizations",
                     "xray:BatchGetTraces",
                     "xray:GetTraceSummaries"
                 ],
                 "Effect": "Allow",
                 "Resource": "*"
             }
         ]
     }
     

   • Assign the role to Datadog in the integration settings.

After successful integration, our AWS account appeared as “connected” in the Datadog integration page.

Datadog AWS Integration page showing the connected AWS account.

Step 2: Enable CloudWatch Logs Forwarding

To send AWS CloudWatch logs to Datadog:

1. Create a Lambda Forwarder

   • In Datadog, we navigated to Logs → AWS Logs in Datadog
   • Followed Datadog’s deployment guide to launch the Lambda Forwarder using AWS SAM or CloudFormation

2. Subscribe Lambda to CloudWatch Log Groups

   • we navigated to AWS CloudWatch → Log Groups.
   • we selected a log group → Actions → Subscribe to Lambda.
   • we chose the Datadog Lambda Forwarder.