Differences between revisions 28 and 29
Revision 28 as of 2007-08-11 02:09:32
Size: 12400
Editor: beast
Comment: reorganized to better reflect current progress
Revision 29 as of 2007-08-15 08:23:33
Size: 12477
Editor: beast
Comment: added yeast
Deletions are marked like this. Additions are marked like this.
Line 46: Line 46:
|| filth || NodeCedarHills         || `filth.cedarhills.ptp` || || filth || NodeCedarHillsCrossing || `filth.cedarhills.ptp` ||
Line 55: Line 55:
|| yeast || NodeRedWing || `yeast.redwing.ptp` ||

Personal Telco VPN

TableOfContents

Overview

This is a project which aims to integrate ["VPN"] technology with ["PTPnet"], focusing primarily on permanent IP-over-IP tunnels created with ["OpenVPN"]. Some history and background are available, as are references to configuration data. Keep in mind the instructions here are only geared towards systems running DebianLinux; someone should probably add notes for other types of node.

Goals

Aside from being a cool idea and a fascinating problem domain from a technical perspective, there are a couple of practical benefits to the project.

  1. Simplified maintenance - Some nodes are trapped behind unfriendly routers doing cone NAT, which prevents the NetworkOperationsTeam from working on them in the usual way. One example is NodeLuckyLab, but there are more than a couple of these out there: NodesBehindNat

  2. Universal connectivity - it would be ideal if all of our different locations were connected, via radio, laser, fiber, Ethernet, frame relay circuit or whatever you like, forming one big ["PTPnet"] cloud. Unfortunately, the fiber-backed wireless dream mesh isn't quite blanketing the world yet. However, in the meantime we can achieve a similar effect with tunnels.

    • One related idea would be allowing more users from outside our network to tunnel in, participating as VPN clients. Think PicoPeer.

      • Technically, quite easy to do, with the foundation that is already in place. It could be set up a number of ways.
      • Several tunnels exist but currently they are all between nodes and specific designated servers; anyone can access the network but only if they are physically present at one of the integrated nodes, which needlessly limits the potential usefulness of the network.
      • What about connecting networks that are not nodes? For example, a home with no wireless network, or a block of servers at a company..
  3. Redundancy - More connections mean more bandwidth for everyone. Even tunnels have the potential to supplement direct links. For example:

    • Additional bandwidth could be gained in situations where multiple routes through different interfaces are available.
    • Fault tolerance is also possible; traffic can be redirected to another path if one interface fails, reducing or even eliminating service interruptions..
  4. IPv6 deployment - Tunnel brokers are effectively the only way that most people in this area can obtain significant IPv6 connectivity. It's better than none at all but these tunnels tend to be unreliable and often suffer from high latency. We have especially if we establish a BGP peer relationship or two at the Internet border rather than falling back to another broker tunnel.

  5. Education - Tunnels give us an opportunity to start acquiring practical knowledge about how to deal with increasing scale in a wide area network, which is going to be invaluable as we begin facing those problems with physical networks.

    • Similarly, we could potentially get a head start on exploring and building potential applications to run on these networks.

Status

The OLSR httpinfo plugin is running on the server: http://donk.personaltelco.net/olsr/

Also, the current OLSR topology from the server's perspective can be viewed [http://donk.personaltelco.net/olsr_topology.png here]. This image is regenerated every 15 minutes.

Currently, the following clients are configured with tunnels:

Host

Node

["PTPnet"] DNS name

ballista

NodeHawthorneHostel

ballista.hawthorne.ptp

beast

["NodeTB151"]

beast.tb.ptp

bone

CoreServer

bone.ptp

buffalogap

NodeBuffaloGap

buffalogap.buffalogap.ptp

cantos

NodePowellsTech

cantos.powellstech.ptp

chevy

NodeMississippi

chevy.mississippi.ptp

cornerstone

CoreServer

cornerstone.ptp

cycle

NodeUglyMug

cycle.uglymug.ptp

filth

NodeCedarHillsCrossing

filth.cedarhills.ptp

frick

NodeAnnaBannanas

frick.annabannanas.ptp

kong

NodeAnnaBannanasStJohns

none yet

lester

NodeDivision34

lester.division34.ptp

loki

NodeCrowBar

loki.crowbar.ptp

number-one

NodeEcotrust

number-one.ecotrust.ptp

serv0

["NodeTB"]

none yet

thehut

NodeOldTownPizza

thehut.oldtownpizza.ptp

vinge

NodePowellsBooks

vinge.powellsbooks.ptp

yeast

NodeRedWing

yeast.redwing.ptp

zeus

NodeBasementPub

zeus.basementpub.ptp

Other hosts were also working recently but need minor configuration updates to match the current server configuration; see the client.conf example below.

How to Help

These are a few of the things that still need to be done:

  • Compatible OLSR configuration for clients is simple but still needs to be documented.
  • We still need to configure some of the NodesBehindNat as clients.

  • Operating systems other than DebianLinux should be supported, with documented configuration instructions.

  • Eventually it would be nice to have all of the systems accounted for by the NodeAudit configured as clients.

  • People who are not members of the NetworkOperationsTeam should be able to configure tunnel clients.

  • Some of the information on this page should be referenced from or moved to other pages.

Also, the "Goals" section above might provide some ideas for more ambitious projects.

Methodology

Design

For now, we are using one central server (donk) and allowing any number of clients to connect directly over TCP, in a classic hub pattern.

At some point in the future, we will likely to reach an upper limit with this design and we will have to re-evaluate our approach given the options which are available at that point. Given the nature of the system, the resource that is most likely to be exhausted first is server bandwidth.

IP addresses are allocated dynamically by OpenVPN on the server; routing is configured dynamically by OLSR.

Configuration

Currently, these directions assume that you are a member of the NetworkOperationsTeam, or at least that you have sudo access on donk, and that the client system you are configuring is running DebianLinux.

First you must generate a new private key and certificate. The following series of commands should do the trick; be sure to replace you with your username and thenode with the hostname of the client.

ssh you@donk.personaltelco.net
sudo -s
cd /etc/ssl/easy-rsa
. vars
./build-key thenode
cp keys/thenode.crt /etc/openvpn/keys/
cp keys/thenode.crt ~
mv keys/thenode.key ~
chown you:you ~/thenode.crt ~/thenode.key
exit

After this initial setup is done on the server, you're ready to configure the client. Note that current versions of the personal-telco-router package depend on openvpn, so it may not be necessary to install it if you're dealing with a node managed by the NetworkOperationsTeam and everything is up-to-date. In this case, you can skip both of the steps which call aptitude.

ssh you@thenode
scp donk.personaltelco.net:thenode.* .
scp donk.personaltelco.net:/etc/ssl/certs/ca.crt .
sudo aptitude update
sudo aptitude install openvpn
sudo mv thenode.key thenode.crt ca.crt /etc/openvpn
sudo chown root:root /etc/openvpn/*

After the key and certificate have been copied to the client, you should delete them from the server.

Create the client configuration file at /etc/openvpn/client.conf:

client
remote donk.personaltelco.net 1195
proto tcp-client
dev tap
ca /etc/openvpn/ca.crt
cert /etc/openvpn/thenode.crt
key /etc/openvpn/thenode.key
comp-lzo

And finally, start OpenVPN on the client-side:

sudo /etc/init.d/openvpn start client

At this point, if avahi-daemon is installed and running, you should be able to reach donk.local from the client, or thenode.local from donk.

If you like, you can now install the olsrd-plugins package, configure OLSR and join ["PTPnet"].

History

I'd appreciate if others with different perspectives would take some time to add their own background to this section. (KeeganQuinn)

Origins

It is difficult, if not impossible, to give credit to any specific source for the idea; many WirelessCommunities have done it before.

Prior to 2006, several attempts were made at interconnecting PersonalTelco nodes over the Internet, although details are not easily found - and perhaps not particularly relevant, anyway. To be sure, the current implementation is not the first to enjoy at least a marginal degree of success.

Statically configured tunnels and routes have been used in the past, as well as OSPF dynamic routing implemented by means of Zebra and Quagga.

2006

In Summer of 2006, JimmySchmierbach and KeeganQuinn spent several weeks planning and testing a design for a virtual network with the potential to scale to serve the entire city. The results of this project were inconclusive due to problems with software stability, but a complete theory for the design was formulated. The central idea is based on a hierarchy with two tiers, referred to as supernodes and nodes. The basic idea was that all of the supernodes would be connected together with tunnels in a full mesh pattern, then each node would maintain a connection with two or more of the supernodes at all times. Fault tolerance is a central element in this design; actual potential for operation on a large scale remains untested.

Jimmy's original drawings specified the three core servers as the supernodes: cornerstone, bone and alitheia. This page previously stated that the design included the idea of supernodes each being connected to a master node (eg. donk), resulting in a hierarchy with three tiers. That is not correct: donk was never a functional part in the original design, and was never connected to the other systems during this project. It would not have provided any notable benefit in acting as an independent tier; with three supernodes in a mesh in the center and a minimum of two supernode connections per node, all of the routes supported by the system could be maintained if any one server failed.

The design is really impressive in theory, but when the time came to test it out, things didn't work out so well. The VPN clients kept taking naps and the dynamic routing daemons often got confused about the fact that we were running a mesh on a layer over their heads. Sometimes machines on the same switch, side by side, would decide that they'd prefer to talk to each other through a big chunk of Internet. This type of erratic behavior made it difficult to take the project seriously at the time.

2007

Thanks largely to the success of JasonMcArthur and AaronBaer with their wireless mesh network at ArborLodge, KeeganQuinn took another look at deploying a city-wide virtual network in Summer of 2007.

It was now evident that the relevant software had reached a degree of maturity which was so clearly lacking a year before. In addition, the topology was simplified to avoid a myriad of potential issues and to speed the process of initial deployment, although a central point of failure was introduced as a consequence. It is difficult to say with certainty which of these factors had the most effect, but the result was that a reasonably stable virtual network could finally be built.

References

Related notes:

Technical details about the proposed VPN configuration: (not currently being used)


CategoryDocumentation CategoryDamnYouKeegan CategoryEducation CategoryMan CategoryNetwork

PersonalTelcoVPN (last edited 2009-10-01 11:54:05 by JasonMcArthur)