Automate AWS Athena Create Partition On Daily Basis

Automate AWS Athena Create Partition On Daily Basis

In my previous blog post I have explained how to automatically create AWS Athena Partitions for cloudtrail logs between two dates. That script will help us to create the partitions till today. But cloudtrail will generate log on everyday. So I was thinking to automate this process too. For this automation I have used Lambda which is a serverless one. This will automate AWS Athena create partition on daily basis. Your Lambda function needs Read permisson on the cloudtrail logs bucket, write access on the query results bucket and execution permission for Athena.

Read More

Automatically Create AWS Athena Partitions For Cloudtrail Between Two Dates

AWS Athena Automatically Create Partition For Between Two Dates

A WS Athena is a schema on read platform. Now Athena is one of best services in AWS to build a Data Lake solutions and do analytics on flat files which are stored in the S3. In the backend its actually using presto clusters. So most of the Presto functions are support and you can use native SQL query to query your data. You can create partitions to speedup your query and reduce the cost for scanning. Here Im gonna explain automatically create AWS Athena partitions for cloudtrail between two dates. When it is introduced I used this for analyze CloudTrail Logs which was very helpful to get some particular activities like who launched this instance, track a particular user’s activity and etc.

Read More

Automatically Add EC2 Instances to Active Directory Domain – Part 1

Windows Servers are in AWS will show some glitches in sometimes. My previous article explains how Windows Server 2016 had some issues with DNS Suffix and Forwarders. This time I got a chance to play around with PowerShell automations. The requirement is automatically add EC2 instances to Active directory domain during the instance launch. It might be an On Demand purpose ec2 or launched by an Auto scaling group.

Read More

How To Migrate PostgreSQL Users To AWS RDS PostgreSQL

How To Migrate PostgreSQL Users To AWS RDS PostgreSQL

According to PostgreSQL migration, we can migrate the databases and tables. I was migrating a Postgresql environment from EC2 to RDS Postgresql. I have completed the data migration. In many blogs, they have mentioned the steps to migrate PostgreSQL users to another server. But in RDS I couldn’t able to do that. I got the below error.

  Read More

Why Windows Server 2016 In AWS Unable To Resolve Public And Local DNS

Windows Server 2016 In AWS Unable To Resolve Public And Local DNS

Before reading further about this DNS issue, I want to clarify few things.

  • This issue happened only in AWS.
  • Windows Server 2016 has this issue.
  • On Domain Controller, I was Able to access the internet before configuring the AD/DNS.
  • On Member Server, I was able to access the internet before joining the domain.
  • Standalone Windows servers are working fine. Only the Domain Controller and the members of the domain having this problem.

Read More