AWS: Prevent VPC Modifications

If you have a busy AWS environment accessed by multiple developers, you will have someone modify your some aspect of your core infrastructure inadvertently.

In our case, we have our VPC-related infrastructure deployed using Cloudformation and maintained via CF stack updates. When devs modified VPC-related resources by circumventing CF stack updates, they rendered our infrastructure out-of-date and un-update-able by CF. Tracking these changes via CloudTrail and rolling them back manually was starting to cost us time and frustration.

Note: Our devs use SSO to login to AWS. Upon login, they assume cross-account roles attached with policies that determine what they can or cannot access.

Assuming that you have your developers sign-in in a similar fashion, below is a policy you can attach to that role to prevent them from modifying VPC-related resources.

Notice how, at the end of this policy, is a section that denies the deletion of this policy from the role? That is key to prevent devs from simply removing this policy from the role.

    "Version": "2012-10-17",
    "Statement": [
            "Effect": "Deny",
            "Action": [
            "Resource": "*"
            "Effect": "Deny",
            "Action": "iam:DeleteRolePolicy",
            "Resource": "arn:aws:iam::999999999999:role/Dev-Role-MBHPPM0DPW90"
Posted in Amazon Web Services, DevOps | Tagged , , , , , , , , , | Leave a comment

Cloudformation: Optional Resource Parameters

When creating Cloudformation templates, occasionally, you come across situations where you only want to remove parameters from a Resource when on certain conditions. As an example, for an ECS Service resource, the parameters ‘LoadBalancers’ and ‘Role’ are both required only if you want your service to be serviced by a loadbalancer. However, you may not want to create two templates to serve these two use cases: 1. Service with LB registration, and 2. Service without LB registration.

In such cases, you can make use of Cloudformation’s conditions, conditionals, and pseudo parameters. This use-case is defined in AWS documentation, but it can be hard to end up on that page via a Google search for terms such as ‘Cloudformation optional parameter turn off’

Here’s a quick example to achieve this in Yaml:

    Type: 'AWS::ECS::Service'
      - LogGroup
          - CreateTargetGroup
            'Fn::Sub': '${ClusterStackName}-EcsServiceRole'
          - !Ref "AWS::NoValue"
      TaskDefinition: !Ref TaskDefinition
      DesiredCount: !Ref AppDesiredCount
          - CreateTargetGroup
            - TargetGroupArn: !Ref TargetGroup
              ContainerPort: !Ref AppContainerPort
              ContainerName: !Ref AppName
          - !Ref "AWS::NoValue"
      Cluster: !ImportValue
        'Fn::Sub': '${ClusterStackName}-ClusterName'
        - Field: 'attribute:ecs.availability-zone'
          Type: spread
        - Field: instanceId
          Type: spread
Posted in Amazon Web Services, DevOps | Tagged , , , , , , | Leave a comment

AWS: Deleting Old Access-Key/Secret-Key Pairs

If you have a busy AWS environment with access to multiple developers , it can be useful to automatically clean up IAM user Access Keys every so often for security.

Here’s a simple Python script that can be plugged into an AWS Lambda function to cleanup Access-Key/Secret-Key Pairs older than 90 days.

The script has a whitelist capability if you want to avoid cleaning up IAM users from a certain IAM group.

The script also removes password profiles from IAM users in case your company policy is to use SSO and prevent users from creating their own AWS Console logins.

import boto3, sys, datetime, time

def cleanup(user,iam_client):
    response = iam_client.list_access_keys(UserName=user)
    for key in response['AccessKeyMetadata']:
        create_date = time.mktime(key['CreateDate'].timetuple())
        now = time.time()
        age = (now - create_date) // 86400
        if age > 90:
            print "AK [",key['AccessKeyId'],"] for user [", user, "] is older than 90 days. Deleting..."
            response = iam_client.delete_access_key(

    # Check if user has password profile
        response = iam_client.get_login_profile(UserName=user)
    except Exception as e:
        if 'NoSuchEntity' not in str(e):
        print "User [",user,"] has password profile. Deleting.."
        response = iam_client.delete_login_profile(UserName=user)

def handler(event, context):
    iam_client = boto3.client('iam')

    response = iam_client.list_groups()
    for item in response['Groups']:

    if whitelist_group_name not in group_list:
        print "Automation Users Group Doesn't Exist! Script Exiting."

    response = iam_client.list_users()
    print "----------------------------------------------"
    for item in response['Users']:
        user = item['UserName']
        response = iam_client.list_groups_for_user(UserName=user)
        if response['Groups']:
            for group in response['Groups']:
                if group['GroupName'] == whitelist_group_name:
                    print "User [",user,"] is an automation-user. Won't be touched."
        if is_automation_user==True:
            print "----------------------------------------------"
            print "User [",user,"] is a regular user. Checking credentials.."
            print "Cleanup on user [",user,"] is now complete."
        print "----------------------------------------------"
Posted in Amazon Web Services, DevOps | Leave a comment

Real Backups On The Cheap

So you have your data on the “cloud” – on Dropbox or GDrive folders – and you believe you’ve done a decent job of safe-guarding your precious files, while really you’ve only saved your files against total computer or hard-drive loss. I used to be this guy until one day when I discovered a few of my precious files went missing from Dropbox. I searched and searched everywhere to discover I’d truly lost them. Was it an accidental delete, or was it a bad program that deleted it? I’ll never know.

While free-Dropbox comes with free 30days restore, it did not have my precious files, and that made me realize that we all have way too much stored in our cloud-folders and there is absolutely no way one can keep a handle on what was deleted/added/modified and when.

Although this incident taught me a lesson and made me store offline-copies of my data on another hard-drive, I really had to go through another painful loss before I seriously started looking for an alternative. Several albums belonging to my really precious music collection (synced across my computers using Resilio, Play Music, and duct-tape) got corrupt and/or missing at some point and were nicely synced across all devices. Again, the sheer volume of data (10,000 music files) ensured I’d find out much much later after too much damage had been done.

So now, I seriously started looking for a solution that would endure the tests of time, stupidity, bad software, and a drunk-me. A solution that would ensure extreme durability while still maintaining relatively quick access when needed.

Enter: AWS S3

Now, I’m sure you know everything about S3 and how inexpensive it can be (0.023 per GB/month for standard-class) to store multiple GBs or TBs of data. But if you’re not an enterprise user, and you’re like me who likes simple and cheap, you’re probably considering S3’s infrequent-storage class at 0.0125 per GB/month. At around 60GB of potential data to store on the cloud, that is just 9$ USD per year vs S3’s standard-class ~17$. But, wait, there’s something even cheaper.

Enter Glacier – the cloud-tape solution from Amazon. Glacier is cheap, as durable as S3 standard-class (99.999999999% durability) and comes at a dirt cheap 0.004$ per GB/month cost. That really is 0.4cents. For 60GB, my cost for the entire year is 2.88$. I remember paying 100$ for a 4.75GB HDD way back. Today, I pay 4.8$ for 100GB of ultra-durable and multi-AZ-replicated storage on Glacier. Times have changed, indeed!

However, before you close this tab and proceed to backup your files into Glacier using a glacier tool (such as, Freeze or FastGlacier), I’d like to let you know that once you upload to a Glacier vault, you will need to request AWS to have access to your data. This includes listing of the contents, so uploading to raw Glacier is not recommended for this use-case.

Instead, I recommend uploading your data into an S3 bucket with a lifecycle management policy set to move data into Glacier 0-days (zero-days) after upload. This ensures that your objects in S3 are moved to Glacier end-of-zeroth-day so you’re not billed for even a single day of S3 storage. AWS moves your data into what it calls the Glacier-class of S3 storage.

This approach ensures that you always have the ability to list your glacier contents using S3 APIs/AWS console. This also lets you use cheap or free S3 tools (such as CyberDuck or S3browser) to upload your data into Glacier vs. having to spend $30+ on Glacier-specific tools such as Freeze.

I hope this was helpful. So far, I’ve managed to upload 20 of 60GB of my data and have been pleasantly surprised by how easy it has been and how much stress it takes off your mind regarding your backups.

A future blog post will detail out the steps required to download your S3-Glacier backups to your computer in an emergency.

Posted in Amazon Web Services, DevOps | Tagged , , , , , , , , , , | Leave a comment

AWS S3 Bucket Policy to Only Allow Encrypted Object Uploads

Amazon S3 supports two types of encryption (server-side-encryption or SSE) for security of data at rest — AES256, and AWS/KMS. AES256 is termed as S3-managed encryption keys [SSE-S3], whereas, KMS is termed, well, SSE-KMS where in the customer manages their encryption keys. A default KMS key is created for you the first time you use a service, such as say, S3.

For more information on SSE-S3, check out this link.
For more information on SSE-KMS, check out this link.

There is growing support among tools (such as Logstash) for AES256-based SSE, so it may make sense to choose this encryption algorithm for your data.

If you want your users (whether IAM users, IAM roles, or regular console users) to never upload un-encrypted data (for, well, security reasons), then it makes sense to have a bucket policy to explicitly deny uploads of un-encrypted objtects.

This example bucket policy was derived using this page . This policy allows for both SSE-S3 and SSE-KMS based encrypted objects while denying everything else.

    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-test-storage-bucket/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": [
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-test-storage-bucket/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"

And, that is it. As simple as that!

If you’re using the AWS CLI to upload objects, here’s how to use various forms of encryption (or not):


#aws s3 cp --sse AES256  file.txt s3://test-dh01-storage/file.txt

AWS/KMS using the default aws/s3 KMS encryption key

#aws s3 cp --sse aws:kms file.txt s3://test-dh01-storage/file.txt


#aws s3 cp file.txt s3://test-dh01-storage/file.txt


Posted in Amazon Web Services, Tech. | Tagged , , , , , , , , , , , | Leave a comment

DC/OS Exhibitor on S3 – Issues & Workarounds

If you want basic resiliency around your DC/OS master nodes when hosting them on AWS, you’ll want to have Exhibitor store its data in AWS S3. In order to do so, you’ll want to grant S3 IAM roles to your master nodes so they may talk to a specific S3 bucket. Then, you use this genconf/config.yaml config to install (or re-install) your DC/OS cluster:

exhibitor_storage_backend: aws_s3
aws_region: us-east-1
exhibitor_explicit_keys: false
s3_bucket: <bucket-name>
s3_prefix: my-dcos-exhibitor-file

Note: You do NOT need to have your master nodes use a load balancer (master_discovery: master_http_loadbalancer ) for discovery even if you decide to use S3 for exhibitor backend. Yes, they’re often used together, but it’s not mandatory to use them together.

Back to your installation, assuming you follow instructions here to continue the installation. Once everything is deployed, head to your S3 bucket, search for and open up the my-dcos-exhibitor-file. Here, two things can happen:

  1. You find the file doesn’t exist. In this case, take a look at your genconf/config.yaml file and count the number of master nodes that you listed in there. Here’s my list of masters:

    If you have less than 3 master nodes in your list, then Exhibitor defaults to using a “static” exhibitor backend, and won’t use S3 to store its config. So, use 3 or more nodes and reinstall.

  2. The second issue that can happen is that only one of the master nodes succeeds to write to S3 into the my-dcos-exhibitor-file and you’re left with a broken cluster. Your services ( #systemctl | grep dcos ) will all fail and your postflight will timeout and fail( #sudo bash --postflight ). You may also see tons of “null-lock-*” files hanging out in your S3 bucket:screenshot-at-2016-12-14-154748

    If this is your case, go checkout the my-dcos-exhibitor-file from S3. If you see something like this, there may be something I can do to help:

    #Auto-generated by Exhibitor
    #Wed Dec 14 19:49:59 UTC 2016\:\:

    If you look at the highlighted lines, you may notice something. What happened to your other master nodes, you ask? Well, I don’t have an answer, but there’s a workaround.

    Edit those two (highlighted in red) lines to include all your servers. Make sure to give them an id.\:,1\:,3\:\:,1\:,3\:

    Now, give your entire cluster a few minutes while all master nodes stop being asshats and start to discover each other. Once Exhibitor is happy, DC/OS stops being whiny. All your services will be up, and you’ll soon be able to login to your DC/OS UI.

    I hope that was helpful. I wasted an entire day (well I got paid to do it) trying to figure this out.

Posted in Amazon Web Services, Linux, Tech. | Tagged , , , , , , , | Leave a comment

DC/OS Kill Mesos Framework

You want to kill a Mesos framework but you’ve no idea how? You’ve looked at this page but it still doesn’t make sense? Then here’s what you need to do to kill a framework on Mesos.

In my case, I have Spark running on DC/OS. In this particular situation, I had Spark in a limbo state not running any tasks we were throwing at it. Our resources were at 100% utilization but none of our tasks were really running, although their status said otherwise. After trying in vain to kill using “dcos spark kill” I tried to kill individual drivers using this:

curl -XPOST https://<your-mesos-endpoint>/master/teardown -d 'frameworkId=6078e555-358c-454f-9359-422f1b6026bd-0002-driver-20161203012921-30627'

But, even though it deleted the drivers from Mesos, the drivers continued to run in the cluster. I figured there had to be a better way to do this. And that’s when I decided to kill the Spark framework instead:

curl -XPOST https://<your-mesos-endpoint>/mesos/master/teardown -d 'frameworkId=6078e555-358c-454f-9359-422f1b6026bd-0002'

And that worked like magic. It killed all the running and pending drivers and also killed Spark.

Once that was done, I removed Spark’s exhibitor entry, and reinstalled Spark.

Posted in Tech. | Tagged , , , , , , | Leave a comment