In general, I am in favor of taking small iterative steps towards improving the product and fixing any problems. Small steps are easy to manage, allow for missteps without too much impact and many small steps can lead to big results. But sometimes, you find yourself in a position that you really need to take a big leap to improve the service for customers. Today, I am talking of one of those I did in the past.
I was with Aramiska. Aramiska delivered two-way internet-over-satellite connections to companies in Europe. Back in 2000 many areas in Europe did not have high-speed internet connections. The photo above was our uplink in Liedekerke, Belgium which we rented from Belgacom. I was part of the engineering team. We built all the software to deploy and manage a large scale ISP. The company no longer exists. The accelerated rollout of cable and DSL made the internet-over-satellite solutions obsolete.
The biggest problem with internet-over-satellite is latency. Latency was usually around 250ms. Latencies like that create havoc with TCP-connections and HTTP requests. One of our standard benchmark sites was nytimes.com. I remember that at the time the NY Times homepage had around 50 objects. Even with HTTP/1.1 which allowed for reuse of TCP-connections, it still took 15 seconds to fetch and render the homepage. This was not a bandwidth problem but a latency problem. It was the single biggest complaint from customers; web browsing was too slow. How do you solve that problem?
During an industry show, I found a company called Tellique which was operating out of Berlin. They had developed a solution to break down TCP-connections and turn them into UDP-streams. Also, they prefetched the whole page at the uplink and send everything over before the browser at the other end would know what to request. Sounds great right? Except it only worked on servers and we had small routers. It was developed for point-to-point setups with a satellite connection in between. We needed a scalable multi-point solution.
Together with Tellique, we embarked on a mission to develop a light-weight client version of their software we could run on our satellite routers. It took 6 months before we could start trying it out. Six months is a long time with unhappy customers and we spent a significant amount of money on this solution. I think it was one of the single biggest expenses the company ever with regard to infrastructure software.
The first roll-out was a disaster. Luckily we did it on a Friday night and customer impact was limited. We totally miscalculated the server capacity necessary to prefetch web pages, compress them and send them onwards. I also discovered that Friday nights was porn night when I was following along the requests. The end result was we had to rollback. This is my star project and I was a young engineer in my second job. I was sure I was going to be fired.
But I did not get fired. We ordered a bunch of extra servers and spun them up. We tried again a week later. This time we got it right. I was ecstatic.
Remember the NY Times homepage? It went from 15s to 1.5s. Customer satisfaction went through the roof.
After roll-out, I spent years perfecting this solution. The problem there are many protocols and applications running on top of the internet. There were many small edge cases which broke in small or spectacular ways. But this 6 months project made it possible to use internet-over-satellite connections for web browsing.
In the end, we ended up acquiring Tellique in another life. Their technology together without multi-point network expertise turned it into a winning solution. I still think fondly about that.
Thinking about it today, I wonder how they solved the HTTPS conversion which makes it hard to transparent proxy solution.