Welcome!

Containers Expo Blog Authors: Jamie Madison, Dalibor Siroky, PagerDuty Blog, Stackify Blog, Matt Lonstine

Related Topics: @CloudExpo, Java IoT, Linux Containers, @DXWorldExpo

@CloudExpo: Article

Monitoring and Analyzing AWS CloudTrail Data

Monitoring and Analyzing AWS CloudTrail data from multiple AWS regions

We recently released AWS CloudTrail integration with Logentries - and not surprisingly we've seen a significant uptick in adoption as one of our most popular integrations. My job as director of customer success is to make things as simple for our customers as possible. One question that consistently pops up, is how to collect AWS CloudTrail logs from multiple AWS regions.

We follow Amazon's best practices when it comes to integrating with, and receiving information from, CloudTrail. In short, this works as follows:

  • When configuring CloudTrail, it will write events to a S3 bucket.
  • You can configure Cloudtrail to send notifications to an Amazon SNS topic whenever new log events are recorded.
  • You can get updates sent to an Amazon Simple Queue Service (Amazon SQS) queue, which enables you to handle these notifications programmatically.
  • To configure Logentries to consume your Cloudtrail logs, simply add the URL of the SQS queue to the Logentries/Cloudtrail setup page.

Logentries speaks directly to the SQS queue inside of your AWS account, so an obvious question that presents itself is: If I'm running in multiple AWS regions, how do I get Logentries to pull from all of the regions?

The simple answer: you don't. Make AWS do the work for you!

Following the steps outlined below, you'll be able to monitor and analyze CloudTrail logs from any number of AWS regions all within one Logentries account.

Create an S3 Bucket
If you're new to the CloudTrail setup, the first requirement of CloudTrail logging is that the logs must go "somewhere." In AWS, this somewhere happens to be a S3 bucket which you should create. Simply navigate to the S3 service and select ‘Create Bucket'. By default, all permissions required are given to the bucket - i.e. there is no extra permissions/configuration necessary to configure CloudTrail logging with Logentries.
Screen Shot 2014-08-19 at 12.55.03 PM

Create an SQS Queue in a Primary Region
Next up, we need to create an SQS Queue to allow Logentries to consume your CloudTrail data. Create a new Queue and provide a ‘Queue Name' - default options are fine.
SQS_Management_Console_and_LogentriesCloudtrail_docx

Add permissions to the SQS Queue
Once the queue has been created, the correct permissions must be applied. When adding permissions to the SQS queue, you need to add your full account number/name (officially called the AWS User ARN).

To get the User ARN navigate to the IAM Service, select the user that you want to utilize and click ‘Summary'. When the user is created within the IAM section, make sure that the user has at least ‘Read-Only' access - so that the user has the relevant permissions to read the bucket. The string you need is available under User ARN in the ‘Summary' section and follows this format:

arn:aws:iam::<account_code>:user/

Next add ‘Receive', ‘Send', and ‘Delete' Actions to the SQS Queue(see below):
SQS_Management_Console
Enable CloudTrail in any region, and publish to an SNS topic

Once the above three steps have been complete, it's time to enable CloudTrail in the relevant regions. Navigate to the CloudTrail Service in your AWS Console and turn on CloudTrail. Do not create a new S3 bucket, but instead select the S3 Bucket created in step one above from the drop down menu. Once you've done this, click the Advanced link. For the first region you enable CloudTrail for, remember to include Global Services under Advanced options - this record API calls from any global AWS services such as IAM or AWS STS. Make sure that "SNS Notification for every log file delivery" is checked, and finally, specify a SNS Topic to publish to. A new SNS topic name should be given - and will be created by CloudTrail.

Follow these above steps for each region that you want to collect CloudTrail logs from. NOTE: when adding subsequent regions you will want to exclude Global Services to avoid duplicate log events being recorded for your Global Services.

Screen Shot 2014-08-19 at 2.48.02 PM
Subscribe the SQS Queue to the multiple SNS topics
Once each region has been setup, the last step in AWS is to subscribe your SQS Queue to each newly created SNS topic. Navigate to the SQS Service in your AWS Console and highlight the queue created in step 2 above. Under the ‘Queue Actions' menu at the top select ‘Subscribe Queue to SNS Topic'. Use the ‘Topic Region' drop down to select the region and the ‘Choose a Topic' drop down to select the topic created in the previous step. Hit the ‘Subscribe' button and wait for the confirmation that the queue has subscribed to that topic.

After selecting subscribe make sure to copy the SQS URL from the ‘Details' section on the page.
SQS_Management_Console1

Setup Logentries to Pull data from the SQS Queue
Login to your Logentries account and navigate to your AWS settings area (My Account -> AWS). Select enable CloudTrail, supply your IAM access key, secret key, and SQS URL that you have copied above. Hit Save! Note: your IAM access key and secret key are made available to your when you create a new IAM user and should be stored safely.

Log Data from CloudTrail will begin to stream in within approximately 15 minutes.

Account___Logentries
Sit back and let Logentries do it's magic!
Visit our CloudTrail documentation to see some of the other cool things you can do - in particular we provide out of the box tags and alerts for important CloudTrail events. Have questions or ideas how we can make our CloudTrail integration better? Reach out to me directly at [email protected].

 

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@ThingsExpo Stories
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...