Diaries of an internet soldier of fortune

Webserver stuff

gregoire — Tue, 21 Jul 2009 01:04:00 +0200

Wow... In my everlasting quest to learn new ServerSignature strings when surfing on the web, I used to Telnet_80 some random hostnames I'm used to paying visits to.
Nothing fancy here, really.
Then I became curious about which servers were the most used.

O Hai! I can haz toolbox ?

First thing was to find a set of tools, widgets, plugins, things that suck ram and displays what I'm looking for in the laziest way possible, because this after all has to remain funny.
First tool is telnet. Here's an example:

gregg@centralperk:~$ telnet www.facebook.com 80 Trying 69.63.184.143... Connected to www.facebook.com. Escape character is '^]'. HEAD / HTTP/1.1 host: localhost HTTP/1.1 302 Found Date: Mon, 20 Jul 2009 22:34:18 GMT Server: Apache/1.3.41.fb2 Location: http://www.ocalhos.ocalhost/common/browser.php Connection: close Content-Type: text/html; charset=utf-8 Connection closed by foreign host.
Well, yeah, that's the BOFH way - painfull, h4x0rZ and not eyecandy at all. On top of that, you would forget to set the host string every second attempt, and would get no response just because of that.
Big up here for the facebook sys architects here that found important to mention they had patched Apache. (...to the morons that found important to read their server header, and obviously I belong to that aforementioned category of deviants).
Well, that being said, it is not handy at all, so I found something else.

After messing a bit with buggy expect scripts of my own to automate that, I found this kneat Firefox Extension that does it for you, all the time, and displays the result in your status bar. Me likey !
Now I can't help, whenever I go to a website, I'd take a quick look at what ServerSignature string it is sending in its HTTP headers.

O Hai ! I can haZ statz ?

Now the thing I was interested in was to see some actual stats on what WebServers were used and a few comments on why if available.
Again, the folks from Netcraft - by the way hats off to you folks for the excellent work you've been doing this past decade with NetCraft - have gathered some precious data around that.

The google webserver army

At some point, I came accross this blog post from dotcomunderground.com which is particularly interesting, as it lists all webservers used by most of Google's ASP apps.
Useless, but definitely worth reading :)

Conclusion

As a conclusion, there is some pretty amazing stuff to be learned by just paying close attention to tiny details. Very likely, one would prefer one webserver instead of another for pragmatic reasons, which is the reason why it is always good to know what alternatives you've got. This is how you'll certainly one day evaluate the likes of Nginx, Resin, thttpd... because each one of these might have its specificities that makes it worth using.
Eventually, you'll come across funny stuff, such as people trying to show off by stating their webserver is a KitchenAid2000 running on a Whirlpool IP OS 12.5 to state their l33tness out loud, but if there's one thing for certain: turning your ServerSignature off is a good security measure, and setting it to a fancy value brings attention, so you might not want to stand out in the crowd an get pwnd just for fun of it :)
Oh and yeah, I found particularly funny that USA's NSA's website runs on Microsoft-IIS/6.0 servers. Maybe it's just a part of it, but I can't help thinking that if I wanted a honeypot, I'd use that ServerSignature on it.

You know your blog has become mainstream when...

gregoire — Mon, 27 Apr 2009 00:54:00 +0200

Once a year, when peeking at the user agents landing on it, you notice something like this:
It used to be all Linux, Macintosh with Firefox, Opera and Safari... :'(
Xtra Speshul str33T kredZ to the ZyBorg robot which makes its way to the browsers list, any luck finding porn in here ? :p

Guy Kawasaki's art of innovation

gregoire — Wed, 22 Apr 2009 02:27:00 +0200

Been a while since this video was released, but I came across it while looking up my bookmarks.
I found it very inspiring back then, and still do. The slides that go with the presentation can be found here:
Guy Kawasaki's art of innovation

Some well spent 55mins if you ask me :)

Guy Kawasaki - Art of Innovation

The silent lords of web2.0

gregoire — Sat, 21 Mar 2009 04:40:00 +0100

Since Web2.0 has been the new namedropping combo that made raising millions for selling available brain time a walk in the park, I've been trying to look out for what was not visible and practically making it possible. Beyond the obvious names that make their way to the headlines and get to be mentioned in the press as inspiring, mindblowing, groundbreaking (you name whatever superlative best suits you), I came to realize that there are some silent players that would rather spend time achieving, breaking new barriers in a quite yet very silent way.

The interesting thing about genius minds is that they don't do press releases. They let their achievements speak instead of their ego. For that they deserve street credit. Presumably way more than techie trolls between grease monkeys at the coffe maker in your average Web2.0 systems/dev/network engineering teams.

I've very often dreamed I was responsible for strategic acquisitions in an infinite cash giant company. Who hasn't. Starting from there, I wondered which where the companies I would like to purchase, that were silent but massive symbols of what makes web2.0 possible. For technically illiterate people, Web2.0 is just AJAX. It now goes far away beyond that. Web sites are no more. There are web applications, web platforms. A software interconnect mapping web's applications together through APIs, some sort of social and logical layer mapped over the internet.

As any concept of this magnitude, it all relies on technical bits and pieces. Here's my own private shopping list of companies and products which I think brought major products and concepts that truly symbolizes the essence of web2.0.

I must reckon that when I compare the very little profesional achievements I've had so far to what some of the brands and products listed below actually deliver, it surely makes me comfortable to see that the complex technical stuff is taken care of. All I'm saying here to the CTOs around the world is that you never spend enough time to try and see the hidden part of the iceberg if you don't deliberately try to raise above your daily technical issues. Please read the below, you might not agree with my side of the story, but at least it could be refreshing.

Oh and before I start, I just want to make it clear that I have no shares in any of the below listed Companies. Neither do I absolutely want to work for any of the below companies. I just found it could be interesting to share my view on a few selected names that I think lead the whole Internet 2.0 industry, directly or indirectly. Enjoy!

JQuery

OK, that one doesn't exactly match my sayings in the preamble. John Resig actually gets an awful lot of well deserved credit for his javascript framework JQuery. He's not exactly what I call anonymous, but there's something in his JQuery work that is exceptional. I do not know if you reader(s) -yeah, S between brackets, because I have doubts on my painful writing being very interesting to average blog readers- have always felt like me about javascript, but it has always horrified me. An interpreted language, with the interpreter in your browser, very often no real means of debugging except throwing dummy variables inside... The list is endless in my mind although I might depict it way worse than what it is and has been in the past: barely object oriented, very loose in the grammar, untyped, well I never found anything appealing about Javascript. What this man did (and I usually don't like too much abstraction over one already existing language) is that he made it simple. He brought brillant concepts on top of it.
Just think about it, if we needed explain what Jquery is in the shortest way, we would just say it is a javascript selector $().
There is so much you can do using it that I won't go over it, but amongst over, it makes DOM traversing sooooooooo simple that you can focus on what you do with it, instead of how you are going to do it. This includes selecting and filtering families of DOM objects in the simplest manner, applying batch treatments to them without actually looping, chaining treatments, playing with their attributes.... the list is endless. And the resulting code is smoooooooth and compact.
The community is rich, the plugins are numerous, and it does evolve insanely quickly. Trying to compare it with any other JS lib in the place is really a waste of time: Prototype/Scriptaculous, MooTools, Dojo, YUI, Spry, and some even more funky ones here.
Please take some time and go play with it. At first you get disoriented and don't really see what it brings, but it comes quickly. And then it really gets you addicted.

ZXTM

This one piece of software here is really one of its kind, and a true jewel. That software from ZEUS cannot be described in a simple way. What I could say to only incent you to go and look for what it does is that it is basically a "Multi-purpose application load-balancer, request router and application traffic controller". It comes as a software license, or can be bought within an appliance, and it will sort pretty much any very high leve traffic loadbalancing issues you have, which I would never have thought possible before I saw it in action. It comes in with its own TrafficScript, Java development guides, and control APIs. It can be setup as a cluster and scale straight forward, does request shaping... well the possibilities with this baby is endless.
As a former Network Engineer, nothing has ever pleased me more than Load Balancing being covered by the Systems Guys without them having to disturb me when slacking (yeah, lots of network engineers tend to slack, that is a fact), and thanks a load for that Mr ZXTM !
Basically, each time I speak with someone who's been using ZXTM, I hear the same speech: "now that I have it, there is no way I can imagine any Tiered architecture without it", and I guess that has to mean something. If I was to conclude on Zeus, I'd say it can as much allow you to scale at moderate cost and gain broader control on how your Tiered architecture scales. A must have.

Amazon's Mechanical Turk

Although I have very little faith in Amazon's EC2 cloud offering, I must admit they are very smart people, with an extremely sound approach of what tomorrow's Internet will be made of. I love the way they they have managed their turn from being just an online shop to a technical institution. Amazon Web Services started with something that seemed very simple: Online Storage (namely S3, over the internet). To whoever has been trying to scale up a storage mutualized storage solution, the word "simple" is not the first that comes in mind. Efficient shared storage is very difficult to achieve, and even though I don't have the technical skills to confirm that, I'll stick to the opinions of my fellow Systems Architects when they mention that it was the most touchy part of any large scale hosting project they had ever gone through. When AWS mastered storage, they sprinkled a little bit of Cloud Computing on top of it with EC2. Once this was dealt with and considered stable and adopted by users, they leaned on the delivery stage to finish covering the whole scope of web applications.
What is important here, is not the quality of the product they deliver (it is supposedly decent, but this is not my main concern here, I'll go over that in my next item). What is important here is the methodology they followed to manage the risk in successfully taking their turn in being a turnkey internet solutions provider, that is the big lesson we should learn from them.
OK, I got a bit far away from the initial item I wanted to write about :)
I had been on Amazon's Web Services page many times before, and I never noticed Mechanical Turk. It seems so fresh as a concept that I'm not even sure I can explain what it is without missing a crucial part. The way I get it is that it is an APIzed Workforce Marketplace. Weird eh ? Not that much, the concept behind all that is CrowSourcing, which is to contract idling crowds to perform mostly simple human-only-doable tasks.
To get an idea of what Mechanical Turk aka AMT allows, just go and have a look at this blog post here.
It is of course being copiously criticized (for some very valid reasons, main one is it basically being turned into a sweatshop), but what really surprises me here is: the initiative - it is something that has never been offered before (I just found out Amazon initially used it internally since 2005 !), and the fact that within a highly advanced and technological offering, AWS raises a significantly important point: "There are still some highly repetitive tasks, that machines however advanced they are can't process without any human intervention" together with the huge contradiction of "Humans being piloted by computers" (sounds like Terminator's Skynet to me) . More info, especially the reason why it is being called Mechanical Turk, can be found on this wikipedia article. One thing is for sure: very few internet related projects lead to as much love or hate as AMT did - and I assume this is the true essence of genius.

Joyent

For the past two years Cloud Computing has really been hype, everyone offering their own variably reliable flavor of that technology: Google App Engine, Amazon's EC2, Rackspace's Mosso, MediaTemple's (gs).
One thing I noticed is that the overall quality of Grid Hosting, or Cloud Computing is very variable, and often offered as-is, without any true explanation of what the value proposition there is in using Cloud Computing instead of traditional mutualised hosting. The folks from Joyent have been very succesfull in making a true product offering with it. From what I hear, they are basically the most technically knowledgeable on the topic and the most advanced. Interestingly enough, the whole solution runs on Sun's Open Solaris, Sun Opteron computers and other major technologies from Sun, empowering efficient (and the word is not to be used loosely) virtualization and scaliing tools.
To get a better feeling on how much the folks out there are involved in Cloud Computing, I strongly suggest you read their company blog, and especially this article.That article here mostly translates what I've always thought about grids/clouds/mutualized architectures without having been able to make it that crystal clear - basically if I needed to be convinced about the real reasons that these architectures are a real necessity for the future, it is exactly the speech I would like to get: brilliant technology, brilliant marketing, brilliant minds. Oh and by the way, it seems that Joyent just enabled ZXTM Application LoadBalancers/Routers in their offering, considering I mentioned Zeus earlier in this post, no wonder why I'm impressed by their service !

SUN

With the recent rumors of Sun being on sale, more specifically IBM being interested in purchasing them, I needed to write a few lines about them. First of all, I must admit I've always been fascinated by this company. Since I've been working, there hasn't been a single year where I've not heard someone telling me "Yeah right, this year is finally the year Sun will file for bankruptcy". There also hasn't been a single year where I haven't seen Sun be reborn with a life-changing innovation. Sun it the internet phoenix_. Sun can't die. Sun is meant to lead and serve as an example for future generations of technical gurus, even if this means not being understood at first. Along the years, Sun has demonstrated excellence in manufacturing the finest and most inventive hardware, proven a mentor in leading edge computing technologies and developments. Besides being the pretty much sole defenders of SPARC based architectures, they also push one of the most reliable OSes (and one of the last and most secure/powerful UNIXes), namely Solaris, and have the very good taste of giving back to the Open Source community through Open Solaris, and MySQL, amongst others.
I wouldn't succeed in listing the numerous initiatives that Sun has been originating lately, so please take some time and pay them an online visit. Here are two tiny elements that I'd like to highlight amongst the many things Sun do: I've always been extremely suspicious about Blade hardware. Basically because it used to be a 1/2 Cabinet chassis that consumed the whole power of two cabinets, which no one ever admitted it was ridiculous. Although it is now a mature concept, Sun has pushed the concept farther, offering to mix and match UltraSPARC blades, AMD blades and INTEL blades into a single chassis, which is available nowhere else, and allows you to completely tailor your resulting chassis to your most complex needs: given the difference of characteristics of those 3 aforementioned CPU Manufacturers, I find it very innovative and smart. Add to those their Virtualized and/or Dedicated 10Gbps Eth Express Modules, and you'll get an infinity of variations.
Lastly, I would also like to mention the recent announcement of Sun stepping firmly into the Cloud Computing Business. From what I read here, they position themselves as a leading provider of cloud computing technologies. From hardware to software, storage and network, they cover the whole range of required elements to provide a single vendor Cloud offer, which is a 1st in this industry. On top of that, if you consider the substantial experience they have gained in supporting Joyent since their early stages, I don't see any reason why they would fail in delivering again a great reliable and inventive solution.

The obvious/forgotten ones

There are many more companies and/or projects that I would like to list, but the interesting part of the exercise was to come up with a short list of players. Amongst others, some names which I would/should certainly have quoted: Arista fka Arastra, Memcached, Hadoop, MogileFS.

To close this post, I'll just say hi and thanks to my usual beer/whisky buddies who very often share their highly valuable technical opinions with me at the bar (I'm just the collector here), they'll certainly recognize themselves through their (former) hostnames: scuderia, bar, daffy, lexomil, floyd, Ambivalence, marmotte...

Also, I would highly recommend that you take a look at those guys here. They have been working on amazing Elastic Hosting offer, owned an excellent and well deserved reputation of experts in that area, and are about to release some insanely sexy features on top of it! I've been using them for their professionalism for ages, and have always been delighted to hear about them releasing new products and/or features, have always heard the most superlatives about their flawless support, so hats-off to you folks at Gandi, what you do is awesome !

Random Content Delivery Networking thoughts (Pt.1)

gregoire — Thu, 19 Mar 2009 00:21:00 +0100

Since I'm a marketing guy now, I might as well sink into metaphysical consideration when it comes to the field of expertise that I'm working on daily. Let's try this, I'll go over the different ways of Delivering Content over the Internet and then wander on each one and deliver some random thoughts on my current and past experience. The good thing here is that I've been working pretty much on every side of the story: ISPs, Carriers, Internet eXchanges and even massive Content Networks. I guess I learned a lot from every of those experiences, from end user Access Technologies constraints to ISPs' national backbones, through central/decentralized content platforms finally to CDN provider.

Sit back, get popcorn ready, treat yourself a beer, this might take a few articles :)

Use CDN, you retarded !

As an introduction, this one link here goes out to all you web platform developers around the world - you need to have a read at what this guy says. Well his not any random "this guy", he's Steve Souders from Yahoo!'s web performance team. The article I mention states 34 rules to make your platform efficient, and guess what ? Rule number two explicitly mentions: "use CDN". One interesting fact you might want to know is that, Yahoo!, pretty much the portal that symbolizes the whole Internet, has never been working without CDN. Even in its early stages, it was powered by one of the oldest Edge Proxy Caching CDNs in place (won't name it tho, but that is not the point).

That is a very well know fact: you will never be able to serve as many users from as many locations as possible, from a central hosting facility in your home country. Sorry if it comes as quite a shock, this is raw facts. Whatever your business model is: either you are subscription based and then performance is the driver to get more users from everywhere and keep your local ones (I'm not speaking about features here, let's consider you have the features and content to be successful) or you get money from your advertising partners, which means visibility must be good and as widespread as possible.

Also, if someday your traffic increase (Good for you, you've become successful, you're living the dream), except that your business plan is not fixed yet, meaning you burn more money than you actually get from your platform. Are you going to deploy a network of yours before you can even get decent and realistic revenue forcast, and if you're smart enough to compute a spreadsheet, there is no way you can conclude deploying your own infrastructure and buying raw bandwidth will actually make you save money... Not now (maybe later), not until your platform is finalized and making money. Not until you've had some experience and insights on how painful it can be to run your own infrastructure when your core job is designing a platform and making it interesting to other people, feature-wise. Right now you need to focus on your own job, and network and infrastructure simply isn't. (trust me, I've been there, or trust me not, I've loved convincing some of my previous employers a datacenter, transit, routers, switches were the only way to go). There will be a time where it will be a smart move, but simply not now.

So basically, whenever you traffic increases, you need performance, at the price of commodity. (yeah right, who doesn't want that...)

The flavors of Content Delivery

Like any decent geek Ice Cream, CDN comes in many flavors. Basically, you'll end up in choosing amongst those three:

reverse proxy caching (a la old school)
storage based (origin distributed)
peer to peer

All of those three have very different features sets, low points, costs and cope with a specific portion of your content. I'll try to go over them in detail.

Rule#1: Get to know the value your content assets have

Yeah right. Say you're in the early stages of developing a promising platform, but you don't know yet what your definite business model is (don't laugh yet, the biggest ones are still looking for their business model, so there is very little chance you'll be right on your 1st shot: best example I have is Joost) and one thing is for sure, you don't know yet what slice of your content is valuable, so get technically ready to adapt your delivery methods and costs. This goes through thinking through the below.

Meet Mr Content Director Engine

First of all, you need to consider that there is not a perfect solution for all of your content. It all goes down to your ability to categorize closely your content. Based on the value and needs of performance that you have set on each "class of content" (man I do hate that CoS formalism, because it puts me back to when I explained that QoS was only necessary when you couldn't afford pipes big enough to carry your traffic and had to make a choice of what to drop when shit hit the fan...), you will serve them using different delivery methods. Basically, it is all about you being able to write your own Content Director Engine. Make sure you have a philosophy on conditional writing URLs pointing to your statics whenever you start building your platform. It one difficult thing to do once your platform is live and running so please take a moment to think this through before you have too many users / too much content on your platform, or it'll become one of those technical hassles that ends up in blood, sweat, tears and big fat ugly downtime.
You might want to be able to choose from where to serve your content depending on:

The Network from which the request is sent (I'm saying AS# , which basically is the network of the ISP from which the content is coming)
The Location from where the request is sent (use GeoMapping databases such as MaxMind, or Quova - MaxMind has a handfull APIs including an Apache module here.
The popularity of the ressource you are viewing: something from the short tail should be served better as it is one of the most viewed items.
The monetization value that your item has: plain "stairway faceplant UGC video" being low value, the hottest tail UGC video being the current buzz being high value, licensed creative content being very high value. Basically the long tail being low value, the short tail being mid value and the creative/licensed content being premium value.

One step further in distributing your content:

More than the above, your content director engine can even move your assets to fully owned storage (expensive) to disposable storage clouds for instance... You can even mix and match professional CDNs with centralized transit delivery or even peering, but this will be only when you have a network to play and cry about, I suggest peering is the cheapest and most performing way to serve your local users.

The trick here is to have everything ready to detect new trends of valuable content, and move them to the proper storage and delivery bundle of yours, with the fewest tweaks in your platform. The general guidance here would be that you use CDN by default, and move it to your own internal distribution when it reaches significant revenue levels, and that you can trigger a project to serve it locally by your own technical means - you'll need hosting, edge proxies, a datacenter switching fabric, and routers with local peerings to an IX or Private Peerings with the local ISPs and transit).

Again, maintaining a network architecture of your own is far away from your initial business, which is supposedly content, so whenever you decide to take the step to move some piece of your content to internal delivery methods, please take some well deserved time to think this through: you need to know what the benefit is, because once you've stepped into building up a network, you have responsibilities in the Internet Ecosystem, and you need to maintain it 24/7 as one of the networks being part of the internet: with great power comes great responsibility some may say, so there have to be well established reasons for you to venture out of your core business !

I'll stop for now and will go over the different methods of delivery that I just mentioned, stay tuned for our next episode, fade to black, advertisement coming up :)

The CDN era

gregoire — Mon, 15 Sep 2008 00:35:00 +0200

CDN is -teh- new trend in the internet industry. That is a fact. Here we go, Media meet Internet, Internet meet Media. What you never thought possible happened: the nerdy sysops met the broadcast men in black. Two worlds that everyone deeply thought would never collide. CDN ( Content Delivery Networking ) is todays latest toy for IT deciders and newly born startups, just as MPLS VPNs were a decade ago. When billion dollar CSI:Miami actor David Caruso founds a streaming media company, the uneducated masses start investigating on what CDN is...

Content delivery networking crash course

Back in the past (I really need to stop this intro, I'm already sounding like I'm Vint Cerf...) delivering content over the internet to a non-local audience really was a pain. We couldn't trust Network Service Providers back then. The aforementioned not really being their fault as long haul communications were both technically complex and costly. Some clever guys from MIT, together with some startup named after a translation of Hawaian word "Clever" tought about working this around.

The initial idea is pretty simple. They would host the heavy content of their content provider customers, and deliver it closest to the audience.
Based on the source IP address of the user requesting the content, they would use some internal algorithm to select the most-likely-closest edge node to serve that content, based on two technologies:

conditional DNS lookups
reverse-proxy-cache

First one to select the best serving node based on source IP AS info and GeoDNS mapping (deliberately simplifying things here). Second one for each edge-node to not have to keep all the content all the time. Content in the cache is served directly, content out of the cache either served by redirect to another node, either proxying http to another node.

This way, the hot content would always be present on the nodes, and cold content could be served from a somewhere more central location.

This is what would later be referred to as the short tail effect (a word from the search engine terminology) : when you are a content provider, you have very little of your content that is being viewed a lot (therefore pre-delivered on your edge nodes), and most of your content that can be cold stored for later delivery.

Flash video, the new stake

The mandatory use of CDN came quite naturally.
One day, some very smart person released some software named Flash, initially a vector motion scripted language that enabled the generation of rich animated, event-driven objects (pretty much programs) to be embeded within HTML pages. The day after, every single browser on the face of Internet had the flash player plugin installed.

Some time later, those same very smart folks out there added some video primitives within Flash, that brought a video envelope (FLV files, flash encapsulated video files) to be read while being downloaded from a Flash application on any web page. This technique was given the name of HTTP Progressive Download or Flash Pseudo Streaming. The key factor in all that was that you could pretty much 'stream' any video from a regular Web Server, just using plain TCP80/HTTP protocol. We very soon saw literally thousand video sites becoming the web's favorite destinations.

Problem is, when you're a geeky web developer, you don't really know anything about Network Bandwidth or Hosting, and honestly, you don't really care about it. When you wake up every day and the bandwidth you are consuming has doubled since the day before, you're kinda facing a decent amount of issues:

the hosting company you use delivers your content from one central location, therefore, you might upset some of your 'remote' fans and lock the growth of your audience
working around architecture and capacity issues is not what you want to do
your business model is yet undefined, so massive purchase of network infrastructure and bandwidth would not be that smart at that point of your startup's timeline...

This is when the $CDN sales rep comes knock on your startup garage door (in Califonia, so it definitely sounds like a cliche).
And he tells you some pretty appealing stuff: he will help you focus on your business:

expanding your audience worldwide (you know that monetizing chimera)
developing killer features, some even relying on his newest flash CDN offer.

He'll basically help you save time and money on not building and maintaining any network infrastructure... How nice is that ?

Epilogue: how to make this post sound not-serious-at-all-eventhough-it-is

After a nice and pricey meal in a trendy restaurant downtown, you'll soon figure out this sales rep is the "new guy in town". He knows personally all of your Web2.0 idols (some even are his customers), he's been in every single entrepreneur gathering on the west coast in the morning, drinking Nappa wine with Kevin Rose and/or Leah Culver in the evening ! He says he's even actually entered the googleplex ! He's standing right in the crossroad of Internet and Broadcast media, waiting for you

That's when you know you need CDN. Whenever this guy with the finest mix of ultra-bright smile and Italian pret-a-porter comes to knock on your garage door, that's when you are going to need CDN.
You better prep for this day so you don't sound like a perfect noob when he comes and tells you those tales of the new CDN era !

Until that blessed day comes, I'll try and keep you posted on this hectic industry in future posts, because ...oh, by the way, it is my new job.

Been there, done that... (chimeras of the network engineering)

gregoire — Sun, 14 Sep 2008 17:49:00 +0200

I don't work in internet architecture anymore. Call it growing old, call it going the easy way, on the whole, I was ready for a change, I was offered change, I signed for it. Still, I will under no circumstance give up digging on network tech news in the industry to keep up with sharpening my vision of the technological ecosystem. Although I'm done (for the moment) with being a grease monkey, I need that technical background to do whatever I'm doing right now.
Now that I'm on the other side of the fence, I'd like to share with the very few out there following my nerdy posts some of the frustrations I've been accumulating over my past Internet Engineering years.
This oughta be fun

The below list of chimeras I've crossed on my misc journeys keep on appearing when I chat over a (numerous amount of) pints with my Network Architects fellows out there. Surprisingly, we seem to share a common frustration over numerous amounts of topics, which, despite the years they have been outstanding, don't seem to ever get resolved.

Layer 3 Switching VS {Switching OR Routing}

That one is actually my favorite, I even extend the concept to pretty much any so called "revolutionary" all-in-one electronic gizmo I come across.

Let's take a lively and yet very current example: cellphones. The initial purpose of a cell phones is to provide the user with the ability to place phone calls from virtually anywhere - doesn't get more simple than that. Now try and go to your favorite local phone-shop to purchase one that isn't either mix of the below:

camera
mp3 player
PDA
GPS

What is usually ends-up with, is you buying one shiny/heavy/expensive device that does it all.
One thing is for sure, the more research the phone constructor will do on one of those features, the more it will be detrimental to the initial purpose: PLACING CALLS ! It ends up with a vast range of side effects such as:

battery lasts about 10mins with all bluetooth, wifi, photo (name any other one) function activated.
device won't fit in your pocket
interface requires a PHD in human-to-machine interface
device is heavy as a brick
AND MOST IMPORTANTLY: it does neither camera, nor mp3, nor PDA nor Phone as well as a single use does.

Let's be serious for a moment: a phone that is only a phone has better chances of working as than any multi-purpose device in the same price range. That is a fact, and mainly because having all those functionalities together only highers the amount of bugs that one features causes on the phone feature

The case of L3 Switching is pretty similar.
Before (yeah, I'm leaping back in Y2K here), you had either Routing issues (no too many really, routing has always came up pretty standard) OR Switching issues (in copious amounts.)

The fabulous idea that L3 Switching promoted is to push a routing table into a distributed switching table.
Sounds appealing on paper: less equipments, cheapest per routed port, maximum flexibility, lower operational cost.

Here's a list of collateral issues it brought:

lazy architects sacrifying resilience of the "Access-Distribution-Core" model to a All in one box, and particularly fiance people reading the specs and telling to techs "why do you need two of these to do that, specs say one does it all ?" - It is not about what the kit can do, it is about how you want things to be done.
memory issues... here they are... you know, when one features sucks up all your L3SW CPU, and all the subsequent cascaded side effects ending up in the box process-switching all your traffic. Been there, done that. Abscence of decent function partitioning when it comes to memory soon becomes your worst enemy.
Device code stability: all goes down to the cellular phone example - the more you want to add feature, the more you risk to jeopardize the already existing ones. One would say that cautious devloppments rules this risk out, but it doesn't... When you need to keep up with your competitor's features, you get sloppy, you don't test as much, and your customers tend to become your field tests. Been there.
lazy engineers again, building temporary-but-everlasting designs, made out of VLAN forwarding all over the backbone, that later blow-up when you least expect them. Been there also, tons of headaches to unravel variously dumb and risky designs.

These are only very few examples on what side effects L3 Switching brought. But let's face it: for those of us who went to network engineering schools, they teach us one thing, and they insist on it being the thing we should always refer to: THE OSI LAYERED MODEL.

I'm not a rocket scientist myself, but I can easily understand why keeping a partition between the Access Layer and the Network Layer makes sense. I like to rely on the work of people that have thought this through, not to justify my copious Network Engineer salary in re-inventing the wheel.

Layer 2 Normalization VS Vendor Specific

This is also one of my big time faves.
I told you, when I signed in the Internet Industry, I felt comforting that older people with grey beards had spend time torturing themselves about what the best way to do things were, and that the fruit of all that common grey-matter were strict Norms. I would bless IEEE and IETF people on a daily basis. They just made my job easier, I just had to read.

IEEE 802.3 together with Equipment Vendors spoiled the naive vision I had. To make it short, whenever it comes to Spanning Tree , I start to freak out instantly. I have never been able to interoperate different equipment vendor boxes without having any side effects. Let's face it, plain STP convergence (30s) time is not sufficient nowadays, especially with increasing bandwidth. Whenever you try to configure RSTP, namely 802.1w accross vendors, unless you deliberately test it, you would always face a not so lovely surprise when discovering that it falls back to plain Spanning Tree convergence because of non interop.

Somme call it PVST+, some RSTP, some 802.1w. What I tend to assume is that 802.1w is too hazy to not let vendors implement their own flavour of it, hence it has to be un-interoperable by default. Amongst other Vendor specific issues: where do BPDUs go ? Tagged in a proprietary VLAN ? Untagged ? Why doesn't the PDF mention it if it is vendor specific ?

I came to the unsatisfying conclusion that Spanning Tree was the oddest protocol on earth: whenever you do Layer 2, you need it to prevent loops. Thing is, once you've set it up, you don't have loop issues anymore, you have Spanning Tree issues - what kind of sense does it make ?

Most of the time, it would force you to signe for a single equipment vendor, which from an engineering and financial perspective, is not satisfying at all !!!!!

Dense Line Cards VS Non Blocking Backplane

This one also is a funny one. How many times have you met that guy in a suit, spoiling your morning coffee moment (because the equipment vendor's office is far away in the suburb you know, and the guy needs to drive to your office, and because of traffic, it's easier early in the morning...), feeding you that usual sales pitch: " you know we have the densiest non-blocking chassis on the market ! *blink* *blink*.
Well OK, but if you question the suit guy enough (no offense, some of them have become close friends...) he'll end up with telling you that it is either density or resiliency.

Let me figure this out, because I need translate your sales pitch into usable specs:

if I stuff the chassis up with all densiest linecards, then the backplane turns into not-so-non-blocking ?
those two management cards that I need to purchase to cope with that situation actually can protect each other ? Why that ? Because of the backplane design, and in order to use all the ports to their nominal bandwidth, I lose the Management resiliency feature ?
that shiny 48x1G card here has a 40G attachment to the backplane ? They don't come up in 40x1G ports ? What do I do with those last 8 ports that I've paid as a part of the kit ?
Oh now, if I do multicast, the replication on separate linecards consumes internal bandwidth... so I will never be able to go linerate ?

What a shock, I would have thought that non-blocking actually meant non-blocking.

Enterprise grade MPLS VPNv4 VPNs

Been a while in the enterprise business, and the more it goes, the less I understand how we can pitch such VPNs.
One mutualized Architecture to Provide all of our customer's VPNs. How comforting is that ? It often ends up with someone asking for MPLS rather than asking for an actual solution - MPLS is trendy. MPLS does it all. MPLS is the most secure thing on earth.
OK, first, if you decide to purchase an MPLS VPN, it is mainly because it is cheaper. Else, you'd build your own network on top of rented leased-lines.
You can't decently ask for a fully private (I'm talking resource-wise here) network as long as you ask it to be setup on a mutualized backbone, right ?
If your goal is to go cheap, why don't you just buy some plain residential internet access on all your sites, with more bandwidth than you actually need, and pay a decent engineer to build tunnels on top of if ? It will come up really cheap, with the same amount of features. If you're paranoid, use two separate internet access vendors on each site, residential bandwidth is cheap nowadays.

What ? You say the support with enterprise telcos is premium ? Well sorry to spoil that, but it isn't, try raising a ticket, you'll figure out by yourself.
Now something even more absurd: QoS
What is the use of QoS, except to cope with undersized pipes ? What is the use of QoS ? I hear you: "It brings Quality of Service" ! Well actually, it is just meant so your important traffic is not dropped whenever shit hits the fan (namely when your employees suck on your centraly delivered internet access).

I say QoS starts with sizing the network correctly. Whenever QoS is triggered, you're past quality. You're just trying to limit the damage done. And the more QoS classes the telco offers, the most comforting you find it...
I'd suggest one simple thing: before going after some trendy protocol, you just specify what your needs are, without even taking market trends into account. You sum-up your bandwidth needs, what features are the most important, what are only nice-to-have. Then you go and ask msic telcos what suits your need the best.

I've got plenty of other chimeras to discuss with you folks, but I'm getting sorta tired here... plus feeling helpless. I might write a couple more in a future post, stay tuned :)

Internet Freedom and Nondiscrimination Act of 2008

gregoire — Mon, 12 May 2008 20:55:00 +0200

I came across this link on a mailing list: Internet Freedom and Nondiscrimination Act of 2008
For once, this legal text is clear, and does take into account technical realities. It also succeeds to this one: Internet Freedom Preservation Act of 2008 . After several bad surprises in Europe, and having to admit that EU hasn't yet realized Net Neutrality is a key element in Internet's Future, it is really pleasant to read that at least one country has a decent vision of how things should still be done. Congrats, Mr. CONYERS and Ms. ZOE LOFGREN for such a brief and detailed Bill !

Cogent fun

gregoire — Mon, 12 May 2008 11:56:00 +0200

Content Networks VS ISPs: Round 2

gregoire — Mon, 14 Apr 2008 15:49:00 +0200

I've spent 2007 designing and implementing a content network for a company that dealt massive outbound bandwidth. I won't insist on the exact volumes, nor on the company's name, but the volumes I talk about were more than the smaller ISPs in France.
Back in end 2006, when we thought copious amounts of bandwidth was THE leverage for getting cheaper transit per meg costs, we soon noticed that it would get harder and harder to peer with such volumes.
In our very naive mind, we would have thought that us, content providers where the ones to make plain internet access interesting.

We obviously were wrong...

Some former defenders of Net Neutrality (a once popular concept on the Internet, which they seem to have forgotten now that it is no more valid against the Incumbent) even turned us down and charged us for peering at the almost same price that transit providers did - for a smaller portion of the Internet of course - some called that being ass-raped, I'd rather call that a 'slightly annoying turn of events', call me a naive optimist if you will.

Thing is, you might also have noticed that those ISPs bragging about their brand-new-tier-1 status have very quickly forgotten how anti-symetric there traffic profile was before that peer-to-peer they keep on disclaiming.
Let us be honest for a moment: how can charging 30€ for 24Mbps be a smart business model ??? If one's got to cover it in Transit, this would mean that, in order to remain profitable, those ISPs should pay a maximum 1,5€/Mbps, just to cover for bandwidth. This of course doesn't include capacity upgrades to scale up to the new needs...
Let's get things straight: in my vision of the Internet, an ISP delivers plain access to undifferentiated access to any IP destination, via any port/protocol combination IP can encap, right ?

If not, that's what we call Online Service or web, or mail, or whatever comes packaged with a proprietary browser, tons of proxies and useless/expensive filtering devices. Ask the guys from America Online, even them have figured this out a while ago, as well as Infonie in France. I'm not actually saying being one is better than being the other, all I'm saying is stop pretending to be what you're not your customers aren't that stupid, they'll figure out one day or another.
Oh, and by the way, for those who just wonder why I'm in such a bad mood re this net neutrality thing today, it's just that I just happened to read those:

The thing that really bothered me when I read that, was that big UK eyeballs (i.e ISPs) where known to be tolerant re peering conditions in the past, and were known to take Net Neutrality seriously.
If you out there feel like I do, c'mon and read again grandpa Vint's sayings:
"The Internet was designed with no gatekeepers over new content or services. A lightweight but enforceable neutrality rule is needed to ensure that the Internet continues to thrive."
More here, in his famous speech - see, even Google didn't twist an old man's judgment the other way round, there is actually hope out there - and no offense Vint : Vint Cerf speaks out on net neutrality.

On an equivalent level again, you'll find a very interesting blog post on Torrent Freak again, here, with both sides detailing their views on why to block/throttle peer to peer or not.(re the Comcast-throttling-p2p affair)
So for those made in china pret-a-porter business executives out there that have been nominated to lead big networks thanks to their talent to read dollars where they should read bytes, please take a moment to re-think your vision of the Internet before you're violently proven wrong: this might be more a promise than a warning, but again, I'm all naive and optimistic.

That should be enough for a daily/monthly/yearly dose of whining and lyricism :)

Quick one: Foundry Ironware 3.8 released

gregoire — Thu, 03 Apr 2008 11:37:00 +0200

For those interested in foundry's RX/MLX/XMR releases, a new version has been released, ironware 3.8
I heard loads of people complaining about Foundry's level of features being way below Cisco's, which was initially true. What I noticed is that development on Ironware is pretty damn fast.

Just for information, some important features brought up with the last versions:

Ironware 3.4 :

Support for new POS interfaces: my I remind you that this is a huge step in Foundry's development as their initial business was Ethernet Switches, then came the Ethernet Routers, and now POS routers.

Ironware 3.7 :

new Ethernet trunk configuration philosophy: it is now possible to configure a trunk with a neo interface : this simply means that trunks now have their own logical interface, such as Port Channel Interface Pox/y on Cisco's IOS.
Yet, I've not yet had the chance to test whether this one interface comes with SNMP counters of its own - in the past, one of Foundry's main drawbacks was that it didn't provide SNMP counters for VLAN interaces, nor Trunk interfaces, so you had to manipulate sums of RRAs for graphing, something really annoying if you're, say a cacti user (a decent workaround would be to use DRRAW, which makes it rather straightforward to sum rrds). I would also need to check whether or not access-lists can be applied to those neo interfaces.

Ironware 3.8 :

This ones brings up features such as:
- bpdu guard
- root guard
needless to say those are quite some lifesavers if you're into Layer 2... Seems to me that Foundry are getting themselves a more and more mature OS these days, with a release frequences that keeps accelerating, which cannot be seen as a bad thing. I'll certainly tell you more once I get a chance ge a little hands on those new releases.

Self promotion: Internet 101 prezo

gregoire — Mon, 31 Mar 2008 16:15:00 +0200

This is pretty new to me, but I've given my 1st engineering school course last Friday, at ESIGETEL, a French IT & Network engineering school, which I am proud to be a former student.
Anyways, the prezo can be found in here, careful though: it is in french. I'll certainly translate it sometimes.

This has been a truly interesting both human and professional experience, I'll certainly do it over with great pleasure if the opportunity comes up again.
enjoy!

[ edit 2008-04-15 ]
For those who are more into video presentations, I've had a shot at this course in the form of a video interview with my man Jean-Michel (Oleane, Witbe, does this remind you anything ? well, he's the man behind all that...): here is is, again, provided you can read french...

Google's secret 10G Ethernet Switches ?

gregoire — Sun, 30 Mar 2008 16:39:00 +0200

I recently read an article mentioning Google was actually manufacturing switches of its own, for 10G Server Distribution in their datacenters. I found the article on Nyquist Capital , which actually based their assumption on tracing the massive purchase of SFP+ components in the optics market. From what we can learn, Google has around 450.000 servers in its "Google Grid", spreaded over the many datacenters they own and rent.

Another assumption on those supposedly home-brew switches is that they be designed around Broadcom's BCM8706's platform.
Though it can seem odd that Google is betting on such a yet-non-standard format of interface namely SFP+, we can see that it is the format of optics that Cisco has chosen for its Nexus 7000 platform new generation high density 32x 10G card here - which would tend to mean that this type of 10G very short reach interface is not bound to disappear that soon.
I must admit that I wasn't able to collect as much info as I would have hoped on these switche's architecture: no white papers, no presos, well... pretty much nothing.
If someone from Google comes up to me with some additional info on those, that he's allowed to share, I'd actually love that.
Still, while searching the web for extra info, I came accross this: Arastra, a newly born 10G ethernet switch vendor. Those kits are very dense, implement 10G Eth SFP+ (48x 10GBASE-CR, 1U Rackspace) and seem to match Google's requirements. On top of that, I remember having read somewhere that Arastra's CEO, Andy Bechtolsheim former Sun Executive, one of the initial Google investors, is closely tied to Eric Schmidt - again, this is only hypothetical, but there might be some hidden link between Google's Secret switching appliance and Arastra's Kits. I'll post more as soon as I know more, in the meantime, if you have any info I don't, do not hesitate to share :)
Some interesting links when it comes to Ethernet:

RIPE Scripting Pt2

gregoire — Mon, 03 Mar 2008 11:31:00 +0100

The events

On 24th Feb, 2008, the whole world became unable to reach YouTube (AS36561). What happened, is that Pakistan Telecom (AS17557), who had decided to blacklist YouTube, started announcing one of their prefixes, but in a more specific way (i.e. announcing a /24 within a /20). This would have ended-up armlessly if PCCW (AS3491), on of their upstreams, hadn't re-announced this prefix to all of their peers (PCCW is a pretty big carrier, with quite a few transit customers...). This /24 range contained YouTube's DNS, so that Pakistan Telecom have been blackholing YouTube for about one hour.

The event chronology and analysis can be found here at RIPE and on the excellent Renesys

What do we know ?

Pakistan Telecom's announcement of YouTube's /24 seems a weird solution for blacklisting
PCCW's Has taken AS17557's announce and spread it to all of its peers, why would they do that ?...

And some of the assumptions and conclusions we might take about all this

Pakistan telecom certainly have Nailed the route to the /24 or had it point to a web server stating "YouTube has been blocked blablabla..."

This would have been done by adding a static route into one(several) of their core routers, which I assume had a redistribute static subnets configured in their IGP. Then, it becomes unclear how this ended up as an announcement in the EGP towards PCCW. Was Pakistan Telekom redistributing IGP into EGP ? I find it weird...
This shows that any network shall filter its announces to its exact IRR allocation.

PCCW was obviously not filtering Pakistan Telekom's annoucements to their IRR prefixes.

They should, as a transit upstream provider, operating a worldwide IP Network, use IRR Records to automate IN prefix-filters on its customer ports.

Avoiding that

Filtering your announcements (transit customer)

This is BGP 1-0-1, you will always want to manually control what your announced prefixes are. It is just about maintaining an outbound prefix-filter, that doesn't change that often, so please do it.

Filtering your customer's prefixes inbound

Your local IRR does contain all the info you need to do that. In order to get an AS# and an allocated IPv4 PI, you need to be in contact with your RIR (APNIC for Pakistan Telekom]) who first records your allocations before you can route them.
You can manually get all the routes of a given AS by doing an inverse lookup of the origin attribute for a certain AS, here it is below, for Pakistan Telekom:

whois -h whois.apnic.net -i origin as17557 | grep '^route:'

Which by the way shows that AS17557 sends-out deaggregated routes, which again, is not very good in terms of Routing Table Growth...

Possibly, you want to automate that, so that your filters are updated every night. If you have one single machine (or one per IRR) that does this for all of your customers, there will very likely be a too many connexions issue with the whois server from your robot, which means that in a few minutes time, you'll temporarily be blacklisted as a potential DoS source.

One other way to do it would consist in not using the whois binary, but a persistent connexion on telnet whois.<your-IRR>.net 43 as TCP/43 is the standard port for whois queries - a -k option for keeping it persistent, netcat / nc for piping any command to a TCP bound port. This would look something like this:
echo "-k -i origin as17557" | nc whois.apnic.net 43 | grep '^route:' The above example is a bit dumb, as the only interrest of Netcat is to maintain TCP/43 connectivity towards the host and therefore not get blacklisted. This is when you start to understand that socket programming is going to be necessary. d'Oh! I won't, it never works the way I want, and I do suck a load at coding network stuff.

If you're still into the socket programming thing, here are some links in PHP and Python that might come in handy to you:

an example of PHP / Socket / RIPE script here
using the excellent Twisted event-driven-networking-framework in Python (probably one of the smartest script languages to be born lately).

As a side note, another way of getting all the prefixes announced by an AS is using the peval binary included in ISC's IRRToolSet or the RtConfig binary that builds router filters accordingly to the IRR reccords. I won't detail this method as I've not found it that optimal, but it still is worth a shot.

But...

We are (assuming I am) way too lazy to dive into socket programming

As a mentioned in one of my earlier posts, IRR 'flat database files' are available on your favorite IRR's FTP. Why bother when you can handle the computation yourself by using your favorite language (even LISP if you wanna) to parse a text file ?

What we'll do first is setup a CRON entry in our robot to go fetch those files we need every night. RIPE's ones are located here:

ftp://ftp.ripe.net/ripe/dbase/split/

Effectively, the 'plain ftp binary' can be used together with macros, defined in the .netrc file in your homedir. The following example of my own .netrc file displays 3 macros, to get 3 different ripe DB files:

 1  machine ftp.ripe.net
 2          login anonymous
 3         password greg@grrrrreg.net
 4  
 5  macdef  getAutNumDB
 6          cd /ripe/dbase/split
 7         lcd ~/scripts   
 8          get ripe.db.aut-num.gz
 9         quit
10  
11  macdef getAsSetDB
12          cd /ripe/dbase/split
13          lcd ~/scripts
14          get ripe.db.as-set.gz
15          quit
16  
17  macdef getInetNumDB
18          cd /ripe/dbase/split
19          lcd ~/scripts
20          get ripe.db.inetnum.gz
21          quit

Then, a tiny bashscript will go fetch those 3 files via FTP, and then gunzip them, just modify the destination location on your machine (line 7,13,19 of the .netrc file) if you wanna:

 1 #!/bin/bash
 2 
 3 echo "## script gets ripe.db files##"
 4 echo "\$ getAsSetDB" | ftp -i ftp.ripe.net 1>/dev/null && echo `date "+[%Y-%m-%d | %H:%M:%S].....ripe.db.as-set transfer complete"`
 5 
 6 if [ ! -e "./ripe.db.as-set.gz" ]
 7         then echo `date "+[%Y-%m-%d | %H:%M:%S]..........ERROR:ripe.db.as-set.gz inexistant"`
 8 else
 9         gunzip -f ripe.db.as-set.gz && echo `date "+[%Y-%m-%d | %H:%M:%S]..........ripe.db.as-set.gz gunziped"`
10 fi

And so on for other ripe.db files...
Next episode, we'll deal with those files with $SCRIPT-LANGUAGE :)

IPv9 ? What the ... L0Lz!

gregoire — Wed, 20 Feb 2008 12:08:00 +0100

I was recently browsing wikipedia, looking for some info to illustrate accurately what people had been calling IPv5 (yes, it did exist and still does), when I came accross an article about IPv9.

How come has nobody told me about this ???
... Being at first too lazy to read the whole article, (and I will never say that enough: the more it goes the more we tend to take for granted anything we read on the Internet, especially on wikipedia !). Out of curiosity, I finally ended-up looking up the IETF to find out a little bit more about what should certainly be a "highly advanced evolution" of our good old but yet familiar IPv4 protocol. I got this result: IETF RFC1606.

Let's take a closer look at that RFC:
Title: A Historical Perspective On The Usage Of IP Version 9
What ? this is not even barely new, and I'm not even aware it ever existed ???

OK, let us see how old it is:
Issue date: 1 April 1994
OK... now I'm starting to connect the dots... what I am reading is one of IETF's excellent April's fools :) (remember that IPv4 packet evil-bit field ? another one of 'em). Still a good laugh (provided you're a geeky network engineer, which kinda restricts the audience) here, where you can find all of IETF's pranks.

I especially like this one extract:
The introduction of body monitors as IPv9 addresseable units injected into the blood stream has been rated as inconclusive. Whilst being able to have devices lodged in the heart, kidneys, brain, etc., sending out SNMPv9 trap messages at critical events has been a useful monitoring tool for doctors, the use of the blood stream as both a delivery and a communication highway, has been problematic. Again, "L0Lz !!!"

More seriously now, IPv9 also corresponds to other serious stuff (but still funny if you consider it is 3 versions ahead of IPv6, there has to be some meaning behind that).
To make it short, IPv9 is very often referred to as being a chinese technology, more can be learnt from this article.
From what I understood by quickly reading, IPv9 comes from China (and is supposedly the only country where it is deployed) and is supposed to address IPv4 address space deprecation. It also seems to include an hybrid DNS facility, to manage Numerical Domain Names, cross-compatible with IPv4 and IPv6 DNS, to handle numerical DNS ressources instead of Litteral DNS ressources.
Sidenote: Writing that above paragraph, I just came to realize I mentioned "IPv9 also corresponds to other serious stuff" therefore having serious second thoughts about it...

What I aslo read, is that IPv9 could refer to TUBA, as in RFC1347, aka TCP and UDP with Bigger Addresses , which would be yet another ipv4 address space deprecation solver but in an even more funky way: this one deals with using re-implementing TCP and UDP over CLNS/CLNP (Connection-Less Network Service/Protocol). For those who don't remember, CLNS/CLNP is an OSI Layer 3 protocol, just as IP is, that uses NSAP addresses instead of IPv4 addresses. CLNS/CLNP is a part of the IS-IS OSI generic routing protocol suite, which is very often used to route SDH/SONET supervision addresses (of such ADMs for instance), of the NSAP format.

Before this article gets too boring (and I strongly suspect it already is!), this again teaches me once more that same crucial lesson:
Wikipedia, or any other info from the Internet, is provided as-is. It always does require checking, since there is a strong chance it is either unclear, hoaxy, incorrect...or just irrelevant. This pretty much the only point this useless post should make, by the way ;)

Undersea fibre cuts in Middle East: conspiracy theory ?

gregoire — Tue, 05 Feb 2008 10:06:00 +0100

I'm a big NANOG reader - plus right now, I'm working on a project in Middle East and of course have witnessed many of the recent traffic changes towards those destinations.
The info has been relayed on many sites already, detailing four Mediterranean undersea fiber cable cuts, centered around the Persian Gulf.

Thread has been going on for a while on NANOG, and the more it goes, the more participants make it sound like an evil plot, in which the US would voluntarily cut undersea cables to isolate Iran from the Internet. (brrrrrr...) Still, no noticeable breakdown in telecommunications towards Iran have been noticed, if those cuts were meant to isolate Iran from the Internet, they obviously failed.
Those theories are apparently (need to be conditionnal on that...) also being pushed by Flag, the international carrier and also subsidiary of the Indian mother company Reliance, through an article in arabianbusiness.com and Egypt claims that, contrary to what had been previously officially stated, ship anchor didn't severe the 1st cable.

Flag now displays an update bulletin, available on their homepage, and here
Flag Europe-Asia and SeaMeWe-4 cable maps here.

Now most blogs and info portal mention that it is now common knowledge that the US have recently been working on Special Warfare ops submarines, such as the SSN-23, aka USS Jimmy Carter, apparently designed for special tasks such as, for instance, fiber cable cut. Those theories can be read here and there.
As usual, I would stick to the excellent Renesys' blog articles as they're free of any such conspiracy theory and just display the effect this 'crisis' has had on general internet traffic:

Now if you ask me, the only thing this disruption prove is that some serious efforts/investments need be thought about in order to turn point-to-point cables to rings, this would at least spared us a big off-topic thread on Nanog :)

Winter 2008 Switching collections

gregoire — Thu, 31 Jan 2008 02:44:00 +0100

It had been a while since CISCO hadn't come with any new switching platform. They had been living on 65xx/76xx for ages, without any significant changes nor anything new against their traditional ethernet switching competitors: Foundry being currently considered as the new switching reference, and Force10 as the most aggressive competitor, with their very dense E-Series

Foundry even went into marketing (who would have thought so ???) They certainly judged their supremacy in ethernet switching made this one advertisement riskless. Nevertheless, Foundry's XMR platform was the 1st densiest ethernet routing platform ever. One could object that features were lacking, but still, they got it first.
Cisco got the message and just announced the Nexus 7000 Series platform:
The Nexus 7000 series looks like a very dense chassis (it can contain 32x10GEth port cards), together with a new OS, NX OS. For once, the chassis includes a cable management solution which is really welcome when you reach those densities on a single chassis.

Now Juniper, finding they were left aside, decided to publish, the same week, on their corporate web, a new series of products: EX3200 , EX4200 , EX8200. The birth of the EX Series is a hudge step forward for Juniper, as they were only known in the past to build (very) expensive, high-quality/end, rolls-type-of Routers only.
Whatever came close to switching at Juniper was their recently issued MX960 Ethernet Routing platform, but nothing that had been designed to switch ethernet Frames as a core activity.
From what can be read on their site, the new EX series range from 1U fixed switch to multi-cards chassis, all running under Juniper's renowned JunOS.
It is common knowledge that Juniper knows a great deal about designing carrier class equipments, and a consistent switching offer, if supported by decent a decent pricing policy, can possibly tempt many network architects in a very near future...

EDIT 2008-02-05
Now it appears that plain Layer2 (i.e. less Layer 3 features in a switch) is getting trendy, Force10 just released their new C-Series, which are actually plain switching chassis, just like Juniper's EX and Cisco's NEXUS 7000 platforms. The product page here displays those new chassis.
Funny to see how we've gone from separate machines to switch packets and route packets, back to full featured L3 Switches, and now back to separating Switching and Routing again. As a parallel, I remember Carriers would never use VLANs in the core, and would route every port in a routing dedicated chassis - some years after, vendors decided that a chassis could and should do both, and decided Metro Ethernet was the new black (mainly when Eth 10G became common). With those new products appearing, it just sounds like we're getting back to the trees - time and vendor case studies will tell and will certainly shed more light on my ignorance.

Now if everyone wants to setup one single chassis in their datacenter suite, I would tend to think that we're only creating Single Points of Failure. The only advantage that I see in such dense chassis is the increased ease of management. I would tend to much prefer Juniper's Virtual Chassis feature, that enables you to treat all of your L2 Switches as a single, virtual, chassis instance - which actually looks way more fault-tolerant.
END EDIT

RIPE scripting

gregoire — Thu, 31 Jan 2008 01:43:00 +0100

I suck at scripting... man I really do. I was today looking for means to get all of the aut-num reccords (aut-num definition here) for a given country. in other terms I wanted to find a listing of all networks within a given country, for instance to evaluate the amount of potential networks to peer with.

I figured out I would go and browse the excellent peeringdb, but since it is user generated, I wouldn't find all autonomous systems and more importantly, scopes on which to filter networks were only regions.

So back to square one. The closest RIPE object that has a country field in it is the inetnum object. Then I just happened to remember that there was some obscure db file on the RIPE ftp (namely ftp.ripe.net), I found it here: ftp://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz

...anonymous user, as for any public utility ftp. Well file is kinda heavy once ungziped, 830Mo or so, plain text.

Here's what a basic inetnum object looks like:
inetnum: 195.8.214.0 - 195.8.215.255
netname: DAILYMOTION-200610
descr: DailyMotion
country: FR
org: ORG-DM5-RIPE
tech-c: DM75002-RIPE
status: ASSIGNED PI
mnt-by: RIPE-NCC-HM-PI-MNT
mnt-lower: RIPE-NCC-HM-PI-MNT
mnt-by: NEO-MNT
mnt-routes: NEO-MNT
mnt-domains: DAILYMOTION-MNT
source: IPE # Filtered

there is actually a country flag, but I also need the IP range (you'll understand below, it is my only link to an aut-num), but I also need to take into account that there might be more than 1x description field between the country field and the inetnum field.

I'll ask grep to look into that, matching
country: COUNTRY-CODE and 4 lines above, which is:
grep -i -B4 'country: COUNTRY-CODE' /home/gregg/ripe.db.inetnum > tmpFile1
(as a hint you might want to run it with COUNTRY-CODE upper-case and lower-case, cause I didn't get the same results, even with the -i option...

Then, I want to keep only the inetnum in a separate list:
grep 'inetnum:' /home/gregg/tmpFile1 | cut -d' ' -f9 > tmpFile2

Which eventually gave me a list of all 1st IP within every RIPE range within the country I was looking for.

What I needed now is to get the aut-num corresponding with the IPs within this list. IP to AS# reversing is a common problem over the internet. You'll find many perl modules on CPAN for that, but I just loved the way the guys at cymru do it.

Then I ended up with launching:
netcat whois.cymru.net 43 < tmpFile2 > resultFile3

Whic after tidying a bit the ouput (mainly doing a uniq) provided me with a decent answer :/ There are certainly uncertainties with the grep -B4 part (i.e inetnum matching the country pattern, but with too many description fileds recorded), but that certainly can be tuned.

If anyone has a more efficient way, I'll take it as mine is not so reasonably funky.

That'd be the 1st post

gregoire — Mon, 31 Dec 2007 10:10:00 +0000

I will be posting here and there in this blog, trying to not let it die, which I can't really promise as I'm being quite busy at the moment.

Stay tuned, things might happen that'd be worth reading, and oh, by the way welcome to Diaries of an Internet Soldier of Fortune.