Monitor Amazon S3 Bucket
Monitoring an Amazon S3 Bucket help you to understand and improve the performance of applications that use Amazon S3 bucket. By integrating Amazon S3 bucket with SnappyFlow, you can have insights (Average Bucket Size, Average Number Of Objects, Requests, Bytes Downloaded, etc...) about your S3 bucket. These insights will help you to optimize and use the resources in an efficient way.
This integration is applicable only for Linux machine.
To collect metrics and logs of Amazon S3 bucket, it is necessary to have a Linux VM and IAM Role with read only access to S3 bucket.
Create a Policy to Access Amazon S3 Bucket
Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
Follow the below steps to create a policy in the IAM console.
Navigate to Access Management > Policies
In the Policies window, click the
In the Create policy window, go to the JSON tab
Copy and paste the below-mentioned JSON code into the policy editor
In the Review policy window, give the Name and Description (Optional) for the policy and review the list of permissions
Attach the policy to a dedicated IAM Role for read-only access.
Configuration in Linux VM to Collect S3 Bucket Logs
Customize the S3 bucket log input format
Click here to learn more about Custom Log Parser.
Add the following configuration to the
- name: s3
- name: lua
- name: record_modifier
Bucket - Bucket consisting files, to be set for monitoring
Prefix - Bucket Prefix location for the files to be monitored
Region - Region of the bucket
Files – Comma separated files to be set for monitoring (Accepts wildcard). Files across multilevel sub-dir. To match prefix /sales-data/year=2020/month=05/day=23/status.txt, /sales-data////.txt.
Exclude_Files - Files within prefix that needs to be excluded for monitoring. It accepts multilevel sub-dir.
Ignore_older - Files to be excluded, whose last modified time falls behind given configuration
Interval_sec - Represents the time at which collection of existing files happen
Mem_Buf_Limit - Maximum runtime memory to be considered while reading s3 objects. Refer Unit Sizes - Fluent Bit: Official Manual to know more about unit specification
Refresh_Object_Listing - Represents the time at which , new wildcard represented files to be looked for.
Ignore_older and Refresh_Object_Listing supports
(m, h,d) syntax.
m - Month
Enable S3 Bucket Logs
To start collecting logs from S3 bucket add the following configuration to the
- name: s3-logs
View AWS S3 Bucket Logs
Follow the below steps to view the logs collected from Amazon S3 bucket.
Go to the Application tab in SnappyFlow and navigate to your Project > Application > Dashboard.
You can view the logs in the Log Management section.note
Once the configuration is complete, the logs will be available within the Log Management section.
To access the unprocessed data gathered from the plugins, navigate to the Browse data section and choose the
Document Type: s3Logs.