Programmatically Trigger Travis Builds on GitHub

Our code for this specific project (which is on GitHub) is designed to handle infrastructure creation and rebuilds. For this purpose, we wanted to to be able to trigger our Travis builds for specific branches via simple Bash scripts, based on meeting certain conditions.

Here we look at logging into Travis using GitHub a token and then using this login to generate a Travis authorization token – a token which allows us to run API calls against Travis’ REST API.

First, we begin by creating a new GitHub token – this is a one time activity. At the end of creating this token, we’ll end up with bare-minimum scopes to trigger Travis builds.

To do this, you click on your profile on the GitHub page and click “Settings”. Here, in the “Developer Settings” heading, head to “Personal access tokens” and hit “Generate new token”. Select the following scopes:

git-2

Check the “public_repo” box if yours is a public Git repository. Hit “Generate Token”.

You’ll now be able to copy the token. I’ve snipped the code below, but you’ll see a long 82-character token string.

git-3

No more GUI. From here on, we can use this Github token to get our work done all via command line.

In your script, you’ll login to Travis this way (make sure the machine running the script has Travis package installed):

/usr/local/bin/travis login --pro --github-token <snipped> --skip-completion-check

The “–skip-completion-check” flag is required to ensure non-interactive CLI access when using Travis.

Next, get a Travis authorization token by doing this:

token=$(/usr/local/bin/travis token --pro --skip-completion-check | cut -f5)

Now, set the following string as the body for the upcoming API call. $branch will be the branch of your project on which you want to trigger the build:

body='{ "request": { "branch":"'${branch}'" }}'

Next, go ahead and make the actual API call to trigger a Travis build using the last known successful build:

curl -so /dev/null -X POST -w "%{http_code}" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Travis-API-Version: 3" \
-H "Authorization: token $token" \
-d "$body" https://api.travis-ci.com/repo/yourproject/requests

The -so /dev/null -X POST -w “%{http_code}” portion is specific to our case where we only want the HTTP response code and nothing else. Our script then takes further decisions based on the output of the CURL call.

Well, that’s it! If you have any questions, please drop them in as comments.

Posted in DevOps, Tech. | Tagged , , , , , , , , | Leave a comment

Automating IAM Instance Profile with Ansible

My previous post talked about creating AWS IAM Instance Profiles so you don’t have to save keys on the instances. In this post, we’ll look at using Ansible to launch EC2 instances with IAM Instance Profiles attached to them (you don’t want to do this manually forever, do you?).

(You cannot attach IAM Instance Profiles to existing instances, unfortunately).

Assuming that you already have in place an IAM user that you’ve dedicated for use by Ansible, let’s head to the next step. In order for this IAM user to attach IAM Instance Profiles to EC2 instances, the user will have to be setup with “PassRole” privileges.

Head to the AWS Console and navigate to Identity and Access Management (IAM). Hit the “Users” tab and click on the user, as discussed above.

policy (1)

Then, hit the “Permissions” tab and scroll down to “Inline Policies”, and hit “click here”.

policy (2)

In the “Set Permissions” page, select “Policy Generator” and hit Select.

policy (3)

In the AWS Service drop down, select AWS IAM, and then select “PassRole” from the Actions drop down. Paste in the ARN of the role created from the previous blog post.

policy-333

Click “Add Statement” and head to the next screen for a Summary.

policy (4)

Hit Next to arrive at the final Review screen.policy (5)

Hit Apply Policy to finish.

Your Ansible playbooks can now create EC2 instances with IAM Instance Profiles attached to them. You’re done!

Posted in Amazon Web Services, Tech. | Tagged , , , , , , , , | Leave a comment

Simple Howto: AWS IAM Instance Profiles

For those of you looking to set up applications to run on EC2 instances without having to put credentials on the machines, there is an option. AWS has a great feature for exactly this purpose, and it’s called IAM Instance Profiles.

The IAM Instance Profile feature allows EC2 instances to call other AWS services on your behalf, with no need for setting up keys on the instance. AWS takes care of securing the keys within instance metadata, and also rotates keys regularly. More info here.

In my case, I was trying to setup Logstash to run on EC2 instances, talking to S3 buckets, without having to hard-code keys or upload them to the instance itself. This AWS feature saved the day.

Let’s get started with setting up an IAM Instance Profile. To begin with, login to the AWS Console and head to Identity and Access Management (IAM). Once there, click “Create Role”. Then, enter a name for the role:

role (1)

On the next page, select Role Type as “Amazon EC2”. This will create the necessary IAM Instance Profile in the background, with the same name as the role.

role (2)

Next, attach a policy to this role depending on your use case. In my case I wanted S3 access from my instances, so I selected AmazonS3FullAccess.

role (3)

Click Finish. On the review page, grab the Role ARN for later use. We’re done!

role (4)

Now, to launch instances using this IAM Instance Profile, simply select the IAM Role from the dropdown on the “Configure Instance Details” page.

role (5)png

You’re all set with IAM Instance Profiles!

Posted in Amazon Web Services, Tech. | Tagged , , , , , , , | Leave a comment

TravisCI: Export From Bash Scripts

Let’s say your TravisCI file is cluttering up due to having too many shell / bash commands in the “.travis.yml” file and you’ve decided to move the commands out to a separate shell script. Now, you’ll want to get export/return data out from this script but you don’t know how. There are two ways to approach this.

If your bash script is only expected to return one value, then you may want to call it from the TravisCI yaml file this way:

script:
- export ENDPOINT=$(bash discover_service.sh);

And you’ll want to add this to the bash script you’re calling (exit returns the variable to whatever’s calling it):

if [ $TRAVIS_BRANCH == 'master' ]; then  
  exit $ENDPOINT  
else  
  exit $ENDPOINT
fi

 

However, if your bash script sets multiple variables or is expected to generate a lot of data, you could “source” it from the TravisCI yaml file this way:

script:
   - . ./deploy.sh

I hope that was helpful!

Posted in DevOps, Tech. | Tagged , , , , , , , , | Leave a comment

Learn Scrum in Less Than An Hour

Well, if you’re part of an organization that does not do Agile (Scrum or not), OR an organization that’s in a transformation towards it, OR in an organization that already does Agile but you’re new and do not know what it is, then look no further than this book right here: http://amzn.to/1tiovv9

Scrum: a Breathtakingly Brief and Agile Introduction

This is an amazingly short, crisp, and distilled book that doesn’t waste any time in beating around the bush or thanking their family and friends for their help and patience in writing it. The book quickly gets to talking about the various features of Scrum form of Agile – including: Roles, Scrum Artifacts, Sprints (and what they mean), and, well, that’s almost it! At the end of the read you’d have come out with a very clear foundational idea of what Scrum is. You could choose to further explore each of the topics on your own (Google?) for further information or clarification.

This is also one of those books that you could give out to your team if you’re part of an organization about to embark on an Agile Scurm journey. It’s cheap, apt, concise and will get the team going in a very short time.

I’m a huge fan of short books that can be re-read regularly – I prefer this form over long books where you forget the context of the first chapter by the time you get to the last one. I chose the Kindle version of the book which is under a dollar and is cheaper than a can of soda.

Posted in Reviews, Tech. | Tagged , , , , | Leave a comment

Public IP Address of Private AWS Instances?

Do you want to find out the public IP address of your private instances (those in a private VPC subnet) but don’t know how? An easy way to do this would be to just do a “whatsmyip” equivalent on the private instance such as this:

wget http://ipecho.net/plain -O - -q ; echo

However, there’s an even easier way to figure this out, especially if AWS is your bread and butter. Every private instance communicates to the outside world via a NAT instance or a NAT gateway (yes, this is your ‘duh’ moment). So, all that you need is your NAT gateway’s or NAT instance’s Elastic IP address and you’re done.

In my case, I was trying to allow an ELB on one AWS account to see traffic from private instances on another AWS account. So, I just added an entry to the ELB’s security group to listen from the NAT gateway’s Elastic IP of the other AWS account and I was done.

Posted in Amazon Web Services, Linux, Tech. | Tagged , , , , , , , , , | Leave a comment

TravisCI and Ansible on AWS

If you’re trying to use Travis CI and Ansible playbooks to spin up and configure instances on AWS, you’ll want to do things at some point:

  1. Configure instances on private subnets with Ansible via bastion host
  2. Secure the said Bastion host’s security group to talk to TravisCI

The problem is that Travis CI is a hosted service on the Internet, whereas your instances are sitting in a private subnet inside AWS. How can you make TravisCI use Ansible to talk to these instances? How do you find out TravisCI’s IP address, given that TravisCI launches your build in a new container each time?

The answer lies in Ansible 2.0’s new SSH Jump Host setting. The following lines should be set as variables for the group or host vars. I set it in group_vars/all because that fits my use case currently:

ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o ProxyCommand="ssh -i <private-key-for-bastion-host.pem> -W %h:%p -q ubuntu@X.X.X.X"'
ansible_ssh_private_key_file: '<private-key-for-private-subnet-instances.pem>'

The first variable is appended to SSH commands automatically by Ansible. The problem with Ansible’s documentation is that it doesn’t mention the need to specify the key file for the bastion host, and I spent hours with errors such as these:

debug1: key_load_public: No such file or directory
debug1: identity file /<private-key-file.pub-cert> type -1

It is probably immediately apparent to someone that works with SSH / Ansible day-in and day-out, though.

Next, you should encrypt these private keys with Travis encrypt files command and paste the output of the command into your .travis.yml file, like so:

before_install:
- openssl aes-256-cbc -K $encrypted_23423423_key -iv $encrypted_23423423_iv -in <private-key>.enc -out <private-key>.pem -d

which dynamically decrypts the file during a build into the container’s current directory (which is the directory where it clones your Git repo, for example). If you want to encrypt multiple files, you should encrypt an archive of the files like so: https://docs.travis-ci.com/user/encrypting-files/#Encrypting-multiple-files

Next, let’s talk about securing the security group for the bastion host. Now, you don’t want to open SSH up for the world, do you? But given that TravisCI launches a new container with a different IP each time, how do you open the security group to this container?

The answer is dynamic modification of the security group. This is how I did it in .travis.yml:

- PUBLIC_IP=`wget http://ipecho.net/plain -O - -q ; echo`
- "echo travis_ip: $PUBLIC_IP >> group_vars/all"


script:
- ansible-playbook 01_create.yml

This effectively puts a variable called “travis_ip” into the group_vars/all file that is then picked up by Ansible when it runs the 01_create.yml playbook.

- name: create/maintaing EC2 security group for Bastion host
    local_action:
      module: ec2_group
      name: "{{ bastion_sg }}"
      description: Security Group for Bastion Server
      region: "{{ region }}"
      vpc_id: "{{ vpc_id }}"
      rules:
        - proto: tcp
           from_port: 22
           to_port: 22
           cidr_ip: "{{ travis_ip }}/32"

That’s it! You’re now all set to use Travis CI and Ansible.

Posted in Linux, Tech. | Leave a comment

Defrost Timer – A Look Inside

If the defrost timer on your fridge is a mechanical timer (works like a wall clock, and ticks annoyingly), then sometime or the other you’ll have this thing fail on you. Ours did, but I wanted to be sure it couldn’t be saved, so I decided to pry it open. It isn’t easy to open the timer up. You’ll inadvertently break a tab or two (I broke one), and you’ll have to be careful opening it up, because as soon as you pop it open all the parts fly into the air. One, you’ll have to search all over the room for parts. Two, because you’ve no idea what the parts are, you’ll never know if you should stop searching for more parts. Third, when you do gather all parts and try to put it back in, you’ll have no idea which part goes where.

I scourged the Internet for a close-up picture of any defrost timer to help me put all the gears and parts back in, but I found nothing.

So, to help some people who might be in a similar situation, I post these pictures of my defrost timer and I hope it helps you put things back in the way they were. When I put all the parts back in, my timer started ticking again. I’ve no idea what fixed it, but the ritual of opening and putting everything back in seems to have worked. Click on the pictures for a full view.

 

IMG_2696IMG_2697

Posted in Tech. | Tagged , , , , , , , , , , , , , , | 1 Comment

A Desert Experience – Jaisalmer & Khuri

I was lying on a thin mat and it was the only thing protecting me from the cold sand under it. I had two layers of blankets covering me, yet my body wouldn’t stop shivering. Not being used to not having walls surround my bed, my mind switched to “Stone-Age” mode of “fight or flight” and I was constantly on the look out for “predators” and other things that might pounce on me any minute. I barely slept a wink.

This was a night spent out on the Khuri sand dunes (Rajasthan), courtesy Badal Singh. Badal rents his place out for tourists and charges bare minimum for the arrangements. Badal is one of the few who believe money isn’t everything. He explained how he makes a good living with what he makes in rents and says he doesn’t need to charge more.

At his house, luxuries are minimum. Hot water buckets are arranged on request and on mercy of power gods (electricity department). Water is heated with coils dipped into buckets – they take a long time to do their job. There is no plumbing in most of the bathrooms so you’ll have to carry your own water. There are no sinks either, so brushing your teeth is awkwardly uncomfortable. There are no room heaters, so prepare to shiver if sitting in shade. Also prepare to shiver when having a bath because the bathroom has a window letting cold air in. Drinking water was provided from an earthen pot – the colony receives drinking water from the government once every few days. And then, I had no phone signal either. My phone was now rendered to being a torchlight.

All used vessels (I saw a huge pile) were washed with only a tub of water. Using more water would severely put them at risk of running out – water they store in a concrete tank built under their house.

Staying at Badal’s house made me realize just how many luxuries I was used to. Back at home, I have 24X7 running water for cleaning myself, for drinking and for everything else. I can have extremely hot water at the flick of a button, running water at the sink, backup power, and super fast Internet and instant communication.

Out here, I was stripped off these luxuries. I felt bare, but also relieved. With nothing to do, you start noticing everything else. I noticed how few clothes Badal and his family members owned. Those that they wore were obviously well worn and tears re-stitched. These were people that had to impress nobody – there were no bosses or colleagues, neither were there any Joneses to keep up with. Pots, pans and cups were of various sizes and shapes so I guessed they collected one here, one there, over the course of many years.

The little huts Badal built were made of a mixture of clay and cow dung, taking no more than a few weeks to build after foundation (of sandstones). They were nicely painted in white and red and looked beautiful inside out. Floors open to the sky were simply mud and cow-dung packed tightly. When I stepped on it barefoot, I realized that it was warm and welcoming unlike parts of the concrete floor that were still cold from last night’s chill. Wow!

Unable to bear the cold, I stepped out onto the terrace and out into the sun. I grabbed a blanket and lay out in the sun for an hour happily sunbathing, warming myself, and just casually gazing around doing nothing – just like the cows Badal owned. Why do human beings have to work?

Food served was basic and masalas were surely avoided. I had poha for breakfast with some really delicious ginger-tea served in a pot. Lunch comprised of bajra roti with dal, rice, and a simple cauliflower curry. Delicious!

Later that afternoon, we were hurried out to the desert on camels (We: there were others beside me and my wife that were staying with Badal) We were asked to bring our own water and nothing else. All arrangements were to be available at the site. It is not until you reach the desert that “arrangements” mean nothing but the availability of food and bedding. Back to the very basics.

It took us an hour to reach our site by camel on a ride that was mostly uncomfortable. Our site in the sand dunes was chosen such that it was low lying and behind shrubs – this was to avoid the chilly winds that were out there to freeze us. At the site, we walked around in the sand (still warm) and watched the sun set into the horizon. There was nothing around us for miles (as far as the eyes could see) but for sand and shrubs, shrubs and sand, in changing order. All we could hear was complete silence, occasionally penetrated by our own voices. Sound travels far when not impeded by obstacles.

We were served an early dinner.  Food was cooked over a wood fire and consisted of balls of atta, called “rota” – wheat flour baked directly in the fire, accompanied with dal and curry. The rotas had a beautiful outer crust that tasted great – I was never going to look at wheat flour the same way ever again. However, we soon realized that the sand just doesn’t finish getting into your shoes, and your shirt, it even gets into the food. There was a lot of sand we ate that night, but the taste of the food and our hunger made us ignore it.

Water is even more precious here, so bowls and pans aren’t “washed” anymore. They’re just wiped off clean with, umm, sand. After a round of generous wiping, our hosts ate their dinner in the same bowls.

Not long after we ate, the bedsheets were laid out and thick, heavy blankets handed out to all of us. We were shown our places and we had all the night to us, out in the open.

That night, I gazed for a long time at the beautiful bare sky. The moon was full and had lit up the desert well. Hundreds of stars twinkled at us, and we said hello for the first time in several years. Back in the city’s light-polluted skies, you barely get to see the moon. Here, we had a vast buffet of stars for our delight.

However, with the cold, and being out in the open, I barely slept at all. In the distance, I could hear a camel happily eating away at a large shrub all night, grinding its teeth against the hardened leaves.

Early next morning, we were woken up to the chirping of a few birds (that braved the cold) and the sun slowly rising above the dunes. There were no bathrooms obviously, so we had to make do with doing “it” in the open. It was an uncomfortable experience, given we had lived all our lives s(h)itting inside four walls. It was still horribly cold and my hands froze as I tried to brush my teeth with a bottle of water.

Meanwhile, breakfast was being prepared and we were shortly called out for. On offer was black tea sweetened with jaggery and rotis that were mildly sweet. They were not made of atta – I asked but I now forget the name. I was hungry and I hogged on 2 or 3 rotis before we left the desert to return to Badal’s house on the camels.

Btw, if you’ve never been on a camel ride, I suggest you avoid it. The ride is bumpy and very uncomfortable when the camel’s walking. At speeds greater than walking (say, jogging), your back is thrown 2 inches into the air with every stride. If you don’t end up with a broken back, you’re sure to end up with body pains that’ll last a while. Don’t do it! (But you’ll probably do it anyway.)

 

Posted in Travel | Tagged , , , , , , , , , , , , , , , , , , , , , , | Leave a comment

Lakshmi Misthan Bhandar – A Money Pit

We recently visited Lakshmi Misthan Bhandar at Jaipur (or LMB) after it was recommended by several people both online and off of it.

We were terribly hungry when we went in, so we planned to dive right into the main course and have some roti and curry. We sat down in the restaurant and looked at the menu. One look at the prices and we left immediately. We wouldn’t ever want to pay Rs.300 for a curry and Rs.80 for a (basic) paratha (ever!). And specially not, given the ambience (or lack of it) of the place. This restaurant was a noisy and closely packed place with the looks of a casual restaurant. The thaali (platter) was Rs.540 per person! The restaurant, however, was packed, so people of Jaipur are either too rich or have very little choice when it comes to food.

We stepped out into the LMB snacks and sweets area, and decided to have some snacks / light food and continue our shopping. The snacks area is a big mess with pamphlets and printouts of “special snacks” and “LMB special sweets” plastered all over the wall. Stacks of sweets and boxes piled in every possible place. I’ve never seen a messier eatery before. It is not enough for the owner to understand what he offers, if the customer doesn’t get it, it’s a fail. There was also a lot of confusion among new customers over ordering food items as there was no visible “cash counter” anywhere. After asking, we were pointed to a small area walled with pre-packed sweet boxes that was supposedly the token counter.

There’s a menu card tucked in at this counter where we could finally figure out what was on offer.

IMG_20150207_185253090

 

Most of the items on this card, as you can see, are still quite overpriced. We ordered a Raj Kachori, a samosa, a Mirchi Bada and a Laccha Rabdi (sweet). The Raj Kachori (Rs.90) is a huge pani puri filled with alu + moong dal (cooked) and curd – it tasted alright. The Mirchi Bada (Rs.25) was filled with alu masala and was tasty. The samosa (Rs.20) was a major disappointment as its taste lacked any character – we left half of it. The Laccha Rabdi (Rs.40) was delicious and we savoured it all, leaving nothing.

Later that evening, we ordered a veg cheese sandwich (for our train journey) and packed some sweets for back home. However, when we opened the sandwich for eating later that night, we were shocked to see how thin it really was. And when we bite into it, we realize that there are no veggies in there! Just cheese and bread (for Rs.90!!). The sandwich was a horrible joke, so we then went and ordered more food from the train’s pantry service. We were left feeling ripped off.

The packed sweet box (petha with sugar syrup) didn’t have a tight seal either and started leaking as soon as we left LMB ( we noticed it later and had to do some cleanup of our luggage).

It looks like LMB is one of those places that continues to rake in money and pull in customers based on nothing but the fact that it was established ages ago. The owner of this place has tried to transform the sweet shop into a all-you-can-think-of eatery, serving items from sweets, ice-creams, burgers, pizzas, thaalis, roti-subji, and everything under the sun. When you step in, you can feel right away that this place doesn’t have a heart, that the owner doesn’t care- it is rather very disconnecting from the “person” and only seems to crave the person’s money.

This probably still would have been a value-for-money place had everything on the menu been HALF its price. However, at these horrendously insane prices (and severe lack of quantity for the price), the Lakshmi Misthan Bhandar is a place best avoided.

Posted in Reviews, Travel | Tagged , , , , , , , , , , , , , , , , | 1 Comment