Rundeck Series: Install And Configure RunDeck 3.0 On CentOS 7

Install And Configure RunDeck 3.0 On CentOS 7
Credit: RunDeck

Rundeck is one of my favorite Automation tools. Here we are going to see how can we install and configure rundek on a CentOS server with mysql as a backend. Even I like Jenkins, but as a SYSadmin, I like the Rundeck a lot.You may think like both can do automation. But as per my understanding is,

  • Jenkins – is for Development and CI/CD purpose.
  • Rundeck – Operation related automations
Read More

Archive MySQL Data In Chunks Using Stored Procedure

In a DBA’s day to day activities, we are doing Archive operation on our transnational database servers to improve your queries and control the Disk space. The archive is a most expensive operation since its involved a huge number of Read and Write will be performed. So its mandatory to run the archive queries in chunks. The archive is depended on business use. Many of us need a copy of the data on an archive database to refer later. To perform the archive we can just simply run the delete query with the limit. But we need to run the query again and again until the matched rows count is 0. We can create a procedure to do this in a while loop. I have created one such procedure to archive many tables. 

Read More

Automate AWS RedShift Snapshot And Restore

Automate AWS RedShift Snapshot And Restore

Redshift will help to handle a massive data warehouse workload. I used to manage some redshift cluster in past. Whenever the developers or I wanted to test something on RedShift, we generally take a snapshot and then launch a new cluster or launch it from the automated snapshot. This is fine for Ad-Hoc workloads. Think something like, if your developers want to continually test and run their sample queries on the cluster on daily basis with updated data then there will be a headache for AWS Admins, So I have prepared a shell script for this to mate this process and it’ll send the email alerts when any one of the steps are failed. This script will help you to automate AWS Redshift snapshot and restore. 

Read More

Automate AWS Athena Create Partition On Daily Basis

Automate AWS Athena Create Partition On Daily Basis

In my previous blog post I have explained how to automatically create AWS Athena Partitions for cloudtrail logs between two dates. That script will help us to create the partitions till today. But cloudtrail will generate log on everyday. So I was thinking to automate this process too. For this automation I have used Lambda which is a serverless one. This will automate AWS Athena create partition on daily basis. Your Lambda function needs Read permisson on the cloudtrail logs bucket, write access on the query results bucket and execution permission for Athena.

Read More

Automatically Create AWS Athena Partitions For Cloudtrail Between Two Dates

AWS Athena Automatically Create Partition For Between Two Dates

A WS Athena is a schema on read platform. Now Athena is one of best services in AWS to build a Data Lake solutions and do analytics on flat files which are stored in the S3. In the backend its actually using presto clusters. So most of the Presto functions are support and you can use native SQL query to query your data. You can create partitions to speedup your query and reduce the cost for scanning. Here Im gonna explain automatically create AWS Athena partitions for cloudtrail between two dates. When it is introduced I used this for analyze CloudTrail Logs which was very helpful to get some particular activities like who launched this instance, track a particular user’s activity and etc.

Read More

Automatically Add EC2 Instances to Active Directory Domain – Part 1

Windows Servers are in AWS will show some glitches in sometimes. My previous article explains how Windows Server 2016 had some issues with DNS Suffix and Forwarders. This time I got a chance to play around with PowerShell automations. The requirement is automatically add EC2 instances to Active directory domain during the instance launch. It might be an On Demand purpose ec2 or launched by an Auto scaling group.

Read More