Category Archives: tech

Fedora EC2 HVM AMI

In this blog post, I am going to tell you how to setup a Fedora HVM Image from the official Fedora PV Image on Amazon EC2. In general, this basically covers how to convert a PV image into a HVM image for AWS EC2. This works for fedora, but might not work for other OSes.

I assume you know about AWS, EC2, AMI, HVM and PV. Amazon has been steadily pushing towards using HVM. With the latest round of launches on July 1, 2014, Amazon now shows only HVM Images by default when you go to launch an instance. You now have to search for PV images. One benefit of using HVM images is better access to underlying hardware resources which gives benefits such as enhanced networking.

Fedora has official Amazon AWS EC2 AMIs available at http://cloud.fedoraproject.org/. However presently it only has para-virtualized (PV) Images.

I have been working extensively on AWS EC2 for the last few weeks and have realized that for best performance, we should be using HVM images.

For this particular project, I was interested in the multiqueue block layer, which was introduced in kernel 3.13.

The first step is to spin up an instance from the existing PV AMI. Its not completely necessary, since you just need the snapshot of the AMI. However I created an instance as I needed to make some changes to the image. The existing AMI available from Fedora has Kernel 3.10. So, I had to do a yum upgrade to get the latest available kernel 3.15.

After launching the instance from PV and making changes as per your needs (in my case, sudo yum upgrade -y), create a new AMI using AWS tools or web console, whatever way you are comfortable with.

For the rest of the steps, you need to setup EC2 API tools or the EC2 Command line Tools. I use EC2 CLI Tools.

After the AMI is ready, find the snapshot id used by the above AMI under EC2 > Elastic Block Store > Snapshot in EC2 Console.

or if you have the ec2 api tools setup:

ec2-describe-images ami-id_of_above_created_ami

and find the snapshot id for the ami. It would be something like snap-a12b34cd .

Once you have the snapshot id, you can register a new AMI using the above snapshot.

To register a new HVM AMI using the above snapshot, you need to use the cli/api tools since AWS still doesn’t have this in the webconsole (it might come soon).

ec2-register -a x86_64 -d '3.15.7-200.fc20.x86_64' -n 'Fedora_20_HVM_AMI' --sriov simple --virtualization-type hvm -s snap-b44feb18 --root-device-name /dev/sda1

where

-d is AMI description
-n is AMI name
-s is snapshot id from step 3.
-a is architecture
–virtualization-type is required for making it hvm
–sriov is for enabling enhanced networking , though it might be redundant, not sure.

This would register/create a new HVM AMI based on the snapshot created by the PV Image.

So, in this blog, we discussed how you can convert an existing PV AMI into an HVM AMI, specifically, how to create a fedora HVM AMI from the official PV AMI.

PS: I have made my Fedora HVM AMI public in the east region. So just search for Fedora and you will find it. Feel free to create AMIs of your own and/or copy it over to other regions.

PPS: Wanna know which cool place do I work where we end up playing with the state of the art latest technologies, be it kernels or the latest SSDs? Hit over to http://aerospike.com/careers to join the team!

How to earn Fedora Badges?

Fedora recently launched https://badges.fedoraproject.org, a recognition system that awards badges based upon certain activities that you do within the Fedora Infrastructure Environment.

I have recently been working with the Fedora Infrastructure and came to know about the badges. Needless to say I was excited and wanted some of my own.

First step to be a part of the Fedora infrastructure is to have a Fedora Account System account. You can signup for it at http://admin.fedoraproject.org/accounts/.

Once you have created your account, you should add a secret question to your account. This will earn you https://badges.fedoraproject.org/badge/riddle-me-this.

https://badges.fedoraproject.org/pngs/fas-riddle-me-this.png

Adding your timezone to your account profile earns you the

https://badges.fedoraproject.org/badge/white-rabbit

https://badges.fedoraproject.org/pngs/fas-white-rabbit.png

By adding your ssh or GPG key to your account, you can earn the https://badges.fedoraproject.org/badge/crypto-panda

https://badges.fedoraproject.org/panda/fas-crypto-panda.png

Accepting the FPCA (Fedora Project Contributor Agreement) earns you the https://badges.fedoraproject.org/badge/involvement

https://badges.fedoraproject.org/pngs/involvement.png

To earn the https://badges.fedoraproject.org/badge/let-me-introduce-myself , you need to create your User twiki page on the Fedora twiki. Mine is at https://fedoraproject.org/wiki/User:Anshprat

https://badges.fedoraproject.org/pngs/wiki-let-me-introduce-myself.png

Editing 10 times on the Fedora twiki earns you the https://badges.fedoraproject.org/badge/junior-editor

https://badges.fedoraproject.org/pngs/junior-editor.png

Participating in one of the Fedora meetings in #fedora-meeting in irc.freenode.net earns you a
https://badges.fedoraproject.org/badge/speak-up!

https://badges.fedoraproject.org/pngs/irc-speak-up.png

This is a brief overview of how to earn some of the badges. I will be updating soon with more badges and more details on the the steps mentioned above.

You can see all the badges at https://badges.fedoraproject.org/explore/badges

And the badges I have earned so far at :

https://badges.fedoraproject.org/user/anshprat

Moving from rackspace to digital ocean

I finally moved my hosting from rackspace to digital ocean (hereafter mostly referred to as DO). The reasons were simple – better config for half the price (especially in terms of memory). In rackspace, I was paying $10 a month for about 245 MB of RAM. In DO, I am getting 491 MB of RAM for $5. I had to resort to 5 minutes cron to keep restart httpd and cleaning up the cache to keep it sane on rackspace. Hopefully, things will be better at DO.

I first came across DO through facebook ads. The thing that caught my eyes was the SSD hosting. In my present job with Aeropsike, Inc, I deal with SSD on a daily basis and surely hosting my own blog on SSD was lucrative. Needless to say getting it at half the existing hosting charges was also enticing. I sat on it for a few weeks, finally got around to clear a DO account and left it short of adding my payment details (to search for a discount code). Few weeks later, I went back, added my payment details (sans any discount coupons) and went ahead to create my first droplet. The UI asked the hostname first thing at the top, and then some clicks to chose your OS version. I missed the hostname part first and selected fedora. On submit, the UI gave an error that hostname is missing. A quick scroll up and then the form was all green. DO boasts of 55 seconds to get your droplet up. While I did not actually time it, the experience was definitely faster than creating EC2 in Amazon WS and rackspace as well.

Screenshot from 2013-08-31 23:05:00

Screenshot from 2013-08-31 23:05:20

Its easy to miss the hostname if you scroll right down to the lower part of page where you do the size and OS selection

DO mails over your root password and then you are pretty much on your own. Here also considering am more comfortable with setting up my own environment using terminal, it was faster for me to create users and add my ssh keys than pre generated users etc.

I then quickly did yum install of wordpress to pull in the required dependencies, export and import from my older blog installation and a quick redo post changing the domain (only dropping the database and then import), my new install of blog was up and running. The reason I chose to reinstall the db for wp was that the first time I did install using stg.hackalyst.info/blog/wp and then changing the css and js links later would have been a pain. (Though now looks like wp has a way of specifying alternate install location in the configuration. Will check it out later).

After installing wp, I tried to activate my jetpack and I kept getting the error:

Your Jetpack has a glitch. Something went wrong that’s never supposed to happen. Guess you’re just lucky: xml_rpc-32601
Try connecting again.

Error Details: The Jetpack server could not communicate with your site’s XML-RPC URL. If you have the W3 Total Cache plugin installed, deactivate W3 Total Cache, try to Connect to WordPress.com again, reactivate W3 Total Cache, then clear W3 Total Cache’s cache.

A few quick web searches later I realised its because the DNS name has not yet propagated for the server. I waited for few hours and later it just worked fine.

Another problem I had with the new wp install was setting up the permalinks. On setting up the permalinks in configuration, I kept getting 404. I searched the docs a bit but found the solution in my own older post when I searched for permalink.

http://hackalyst.info/2010/02/17/setting-up-your-websiteblog-using-wordpress-on-a-slicehost-slice/

In short, I had to change
AllowOverride FileInfo

in directory directive in httpd.conf found in /etc/httpd/conf folder.

Rather this time, I decided to add the blog directory itself to the virtual host config and voila, it all worked fine.

Another warning I got while doing the wordpress install and configuration with apache httpd was

AH00548: NameVirtualHost has no effect and will be removed in the next releas

I wanted to know what the change actually meant and found this link in a comment here
httpd.apache.org/docs/current/vhosts/name-based.html which lead me to

http://httpd.apache.org/docs/2.4/upgrading.html#misc

The NameVirtualHost directive no longer has any effect, other than to emit a warning. Any address/port combination appearing in multiple virtual hosts is implicitly treated as a name-based virtual host.

Though I still haven’t found what the number AH0048 mean. Maybe I will have to dig into the source code or mailing lists archive to find the meaning of that number.

Coming back to DO, though they advertise SSD setups, the vm I am on says its rotational.

[root@hackalyst conf]# cat /sys/block/vda/queue/rotational
1

Will see if I can figure out the actual disk.

So far my DO experience has been good. Fingers crossed. Lets see how it goes. I will be disabling my rackspace server soon.

IPv6 is still missing in DO though. So I might get back to tunnel like how I was doing on slicehost before moving to rackspace.

Here is how to get ipv6 using tunnels. Though the blog post says in India, its geographic independent.

screen vertical split rpm

I ve been using screen with vertical split for sometime now. And whenever I move my workspace to a new environment, its a fight to get either a build or a rpm with vertical split.

Recently I moved to centos 6.3 for my workspace usage and used the following rpm for install with glibc < 2.12

ftp://fr2.rpmfind.net/linux/fedora/linux/releases/15/Everything/x86_64/os/Packages/screen-4.1.0-0.3.20101110git066b098.fc15.x86_64.rpm

http://www.rpmfind.net//linux/RPM/fedora/devel/rawhide/x86_64/s/screen-4.1.0-0.15.20120314git3c2946.fc20.x86_64.html

how to create ipv6 reverse DNS entry

Lets begin with what all we will be covering in this blog post.
We will be covering a lil bit of what is a reverse DNS entry and why do we need it.
We will then cover how to create a reverse DNS entry for two types of ipv6 entries:

    1. A 6to4 ipv6 ip using our own dns server and a 6to4 nro delegation
    2. A tunnel broker ipv6 entry using dns provided by he.net

The reverse DNS look up helps to resolve an ip into the respective host name. It is, like the name says, “reverse” of what the DNS normally do.

DNS is used to convert a human readable name like “hackalyst.info” into its corresponding ip.


host hackalyst.info
hackalyst.info has address 50.57.67.195

reverse DNS entry helps to resolve the ip into a hostname.

$ host 50.57.67.195
195.67.57.50.in-addr.arpa domain name pointer hackalyst.info.

Now, why is reverse DNS required?

One of the reasons nicely explained at godaddy is to fight spam.

Other than that, its always cool to have your ip map to your domain name ;)

There are two ways of creating a reverse DNS entry for ipv6.
1) You can create your own DNS server, point your domain name provider to point to your additional DNS.
2) You can use a free dns service provided by dns.he.net

Lets first go through the first option which is slightly lengthier and more DYI and CLI friendly :P

The wikipedia entry on 6to4 pointed my towards 6to4.nro.net that can be used to create a revese DNS PTR for a 6to4 ip, i.e, an ipv6 starting with 2002:.

The 6to4.nro.net needs dns servers with zone for this reverse delegation. None of the dns servers I host with (domaincontrol.com through godaddy, stabletransit.com via rackspace and he.net) were accepted in 6to4 form. So I decided to setup my own DNS server.

First I setup the DNS name for my DNS (yeah, thats required!) at my existing dns provider.
I am setting up my dns on my own server, hence can simply create sub domains (I used dns1 and dns2 instead of traditional ns1 and ns2) pointing to my own domain. You can create such subdomains pointed to the actual hosts where you will be setting up your DNS. You can set it up on one host or on multiple hosts.

I setup my own DNS server using instructions from devshed forums. Its pretty old but still works.

At the end of configurating the dns server, I initially got an error saying:

_default/67.57.50.in-addr.arpa/IN: file not found

That was because the file name in the config and my actual file name were not the same. After I fixed this issue, still named would refuse to start without throwing any erorrs.

An inspection of /var/log/messages said something about unable to write to the log file.

Jun 24 11:58:22 deltacore named[31254]: the working directory is not writable
Jun 24 11:58:22 deltacore named[31254]: isc_stdio_open 'query.log' failed: permission denied
Jun 24 11:58:22 deltacore named[31254]: configuring logging: permission denied
Jun 24 11:58:22 deltacore named[31254]: loading configuration: permission denied

The way to fix was to change the ownership of the /var/named folder and /etc/named to named :)


#chown -R named:named /var/named /etc/named.conf

And then all was fine :)

After setting the above DNS server, I needed to add the PTR info for my ipv6.
I found http://www.fpsn.net/index.cgi?pg=tools&tool=ipv6-inaddr via searching and headed over to create my reverse PTR configuration for my own DNS.

In the form:
Record type : select ip6.arpa (new standard)
Assigned IPv6 Block: (your ipv6 block)
Admin email:
DNS Server: (DNS Created above)
Secondary server (DNS Created above)

Then it gives the configuration file which you can add to your named.conf and create the reverse zone file.
Like it says at the end of generated reverse zone file, you need to add the AAAA entry in your respective zone file.

After this, it was just playing around to add “my” DNS servers to my “Domain” NS entries with my service provider.
My primary DNS provider is godaddy. I just added my own DNS entries into the NS entry of my domain.

If you do not want to create your own DNS server (or cannot due to resource crunch), you can use freely available DNS servers from he.net.

For this second purpose, I am going to use both the 6to4 ip above tied with your domain and another ipv6 obtained through tunnelbroker.

DNS provided by he.net can be used for managing your domains and these are pretty good in terms of usability.
Add your domain to your dns.he.net account, create your A,AAAA and other entries.

For creating your reverse DNS entry, you need to create the PTR record. Its available under “Additional” menu in dns.he.net.
For finding out the value of your ipv6 PTR, head over to http://rdns6.com/ and enter your v6 ip. The nibble value is what basically goes into your PTR record. In dns.he.net, the above nibble would be added as Name in PTR entry and your domain name in the Hostname.

If you are using a tunnel broker ipv6, you can login to the dns.he.net using same credentials as your tunnelbroker and it would import and delegate the ipv6 provided to you to the he.net nameservers. (I might be wrong here as I ve been using dns.he.net for quite some time and do not remember the initial steps I took).

Once you ve assigned dns names to your ipv6 ips, you can go and assign those ips and respective domain in the dns.he.net. This creates the PTR record for those ipv6 pointing to your domain.

at the end of the day, this is what should be the result of a perfectly working PTR record.

$ host hackalyst.info
hackalyst.info has address 50.57.67.195
hackalyst.info has IPv6 address 2002:3239:43c3::1

$ host 2002:3239:43c3::1
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.c.3.4.9.3.2.3.2.0.0.2.ip6.arpa domain name pointer hackalyst.info.

$ host hackalyst.homeunix.org
hackalyst.homeunix.org has address 106.51.119.133
hackalyst.homeunix.org has IPv6 address 2001:470:5:869:21e:c9ff:fe03:803b

$ host 2001:470:5:869:21e:c9ff:fe03:803b
b.3.0.8.3.0.e.f.f.f.9.c.e.1.2.0.9.6.8.0.5.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa domain name pointer hackalyst.homeunix.org.

So now you know how to create reverse DNS entry aka PTR for your ipv6 :)

cheers!

Google/Youtube’s ISP specific cache/tieups ?

hi,

Today we are going to discuss possible google tieups with local ISP’s to create caching layers in order to deliver better experience to high bandwidth customers.

I stumbled upon the same while browsing one of google’s pages which had an embedded youtube video. Initially it was very slow and I was quite surprised since I am on a 10mbps bandwidth connection and am usually able to stream even HD videos on youtube without any buffering delays. Then I realised that most probably I was going over the ipv6 network (and hence tunneling all the way to US and not getting local content) and hence the delay. I fired my wireshark and indeed I was going over the ipv6 network. I mulled over disabling the ipv6 in router and keeping it only in my dev box when I realised that I can simply disable ipv6 browsing in my firefox :).

about:config -> network.dns.disableIPv6;true

And am done :)

After doing the above, I reloaded the earlier page with firebug enabled. The embedded video was now playing smoothly without any buffering delays. And then all of a sudden I saw this in the firebug network panel:

o-o.preferred.actbroadband-blr1.v3.lscache3.c.youtube.com

wait what?!
So google has a cache specifically for my ISP?! (I use ACT broadband).
Now that my curiosity was pipped, I decided to do a lil more digging.

[anshup@mouthwa ~]$ host o-o.preferred.actbroadband-blr1.v3.lscache3.c.youtube.com
o-o.preferred.actbroadband-blr1.v3.lscache3.c.youtube.com has address 202.83.22.14
o-o.preferred.actbroadband-blr1.v3.lscache3.c.youtube.com has IPv6 address 2404:6800:4007:2::e

Oh, so they ve also ipv6 enabled it, nice.

whois 202.83.22.14

descr: BroadBand Internet Service Provider, India
..
address: ACT Television
address: # 29/4, 4th Floor, Trade Center,
address: Race Course Road, Bangalore - 560 001

[anshup@mouthwa ~]$ whois 2404:6800:4007:2::e

inet6num: 2404:6800::/32
netname: GOOGLE_IPV6_AP-20080930
descr: Google IPv6 address block in AP
country: AU

Ah, so while the ipv4 cache was coming from my local ISP, the ipv6 cache was coming all the way from australia? So even though google might ve teamed up and linked a deal with local broadband providers, it still is some miles away in getting ipv6 cache to India? Or possibly because am on an ipv6 tunnel rather, it ended up using my tunnel endpoint as the preferred location rather than my original location.

The more interesting part is that Google has apparently created up local caches at ISP ends to help give better speed to the end user. I need to check it up on some other ISP s here in bangalore and then will update the results here.

cheers

how to get ipv6 connection in India

It is possible to get ipv6 connection in India using tunneling (ipv6 over ipv4). There are multiple providers, but he.net is one of the simplest ones to use and setup. And its for free! You can set upto 5 tunnels with every free account. You’ll mostly need one or two maximum as each tunnel needs a valid ipv4 ip on your end.

To set a free tunnel, head over to the tunnel broker site of he.net.

Register a new account. After successful registration (and confirmation), login, click on “create a regular tunnel” under user functions, provide your ipv4 address and voila (usually its the same as “You are vieweing from” that is shown under the text box for “IPv4 endpoint (your side)” and you are ready to go!

If you want to use google services over ipv6, I suggest you use one of the US endpoints of the tunnel, as it looks like google has whitelisted the US endpoints. The Asian endpoints dont ve access to google and facebook over ipv6.

Once you ve created your tunnel, you need to setup your system. I ve tried it on linux, mac and airport extreme. The example configuration showed in the tunnel page (once created) works pretty well for all that I tried. Most of them need you to copy paste the commands given in a command line terminal.

Once you are done with all the above, comes the most important part. Adding the DNS! Even if you ve a fully functional tunnel, you wont be able to browse or use it unless you ve the DNS configured for ipv6.

In your version of OS, configure the DNS for he.net

2001:470:1f06:b8c::2

Check it up online how to configure a nameserver/dns for your OS.

If you ve a airtel connection, or other ISPs which result in you ving different ips, no worries. You can update your tunnel endpoint (i.e, your ip) in the tunnel configuration easily. Either using the web UI by logging into the tunnelbroker.net site, or using APIs available at http://ipv4.tunnelbroker.net/ipv4_end.php

Usage: https://ipv4.tunnelbroker.net/ipv4_end.php?ip=IPV4ADDR&pass=MD5PASS&apikey=USERID&tid=TUNNELID
-or-: https://USERNAME:PASSWORD@ipv4.tunnelbroker.net/ipv4_end.php?tid=TUNNELID (auto-detect IP)

https://USERNAME:PASSWORD@ipv4.tunnelbroker.net/ipv4_end.php?tid=TUNNELID&ip=IPV4ADDR

Let me know if you need more help with setting up your ipv6 tunnel!

My home network and my blog both are now on ipv6, thanks to the tunnels provided by he.net!

Blogging from mobile

I am kinda late to the mobile blogging scene, owing to slow gprs speed, missing 3g connection and a mobile device with wifi connection (well I had the ipad for about 6 months now but guess I was lazy). So now that I ve 3g on my nexus s (not to forget wifi on it as well) , I ve now installed WordPress for android on it. I can immediately see the pros n cons of it. Pros – blogging on the move.  Cons – the linking to other pages etc would be more cumbersome as you can’t just open a tab, copy a link and paste. A big major pros is the auto suggest and auto correct that you get on all mobile phones. Well… I guess am gonna be happy about it…

UPDATE: I opened the post in my laptop browser to add the android marketplaces link. Guess, mobile blogging will take sometime before its my first choice. I ll try it from my ipad as well. Somehow it appears as if thats gonna be more intuitive.

My hunt for the perfect browser for work :!

UPDATE: And now I have moved (back) to firefox 4! Firefox 4 is fast, so far no crashes and is looking good. more on this later as I spend more time on it!

hi,

First and foremost a little into what this post will give you. In this post, first I talk a little about why I need this post. Then a little of my browsing habbits history. Then some of my problems. And then my problems.

This post started when I realised I ve been switching between browsers one after another whenever I face some issue with one. Now few months down the lane, I don’t remember what the problem was and why I switched away from a particular browser and to what and why. So this post is more like a little documentation effort.

Now a little into my browser usage history. As a kid I remember using IE, Firefox and a little of Opera. In college, I was primarily a firefox user (with a little bit of elinks thrown in whenever the network would be clogged down). After I started work, one of the worst versions of firefox was released. FF 3.0. It would crash like a drunken dog gone nuts on the roads. I switched over to Opera at this instance after suffering for weeks (on fedora). Then I switched to Mac. My love for Opera continued. Ofcourse I have firefox with firebug and webdeveloper and various profiles. And then, I guess it started crashing for some reasons. I tried safari but wasn’t really impressed. At this time I primarily switched to Chrome. I had tried chromium on fedora earlier and Chrome had evolved a lot from its chromium days.

At the moment, am back to Opera. This post is being published in Opera. And the reason. Well, the latest version of Chrome is buggy or maybe its the speeddial extension. Now when I click on any speeddial, the speeddial opens up, but the address bar is empty :!

And its hightime I try Opera 11 :)
In the meanwhile, I ll be downloading latest firefox nightly release aka minefield :)

And will keep this post (or further posts) as a part of documenting my waddle through the various browsers. (IE wont be part much of it, though I do use it once in a while owing to some stupid IE rich web applications.).

More on it later. Cheers
Anshu Prateek

Free Live Streaming TV / News Channel

This is something I picked up quite sometime back when I needed to watch news in office for breaking news. Initially I configured only one channel, ndtv. But when I landed in germany for december vacation, I picked up a couple more as not all channels were freely available here.

So hit http://anshprat.info/news/ and grab your favourite news channel for free online :P Right now it has the following channels:

bbcworld

cnbc

ddnews

ndtv

skynews

starnews

And my favourite watch it all in one screen :P

ALL