Distributing packages
Distributing data can be an issue in FLOSS projects. One day, some IPv6-P2P-multicast-awesome-featured technology will solve all our problems. Internet will be made of undistinguished nodes, and the concept of client and server will disappear. Oops, that’s not my point (for French readers)!
ftp.archlinux.org, the default mirror for Arch, uses too much bandwidth. Even if the installer proposes to change this, it seems that a awful lot of persons don’t use it. That’s why devs chose to limit each connection to 50kbytes/s on it.
I wrote two small PHP scripts to allow geographic, cached distribution of clients to other mirrors.
On the first connection of an IP, it uses GeoIP to choose its “nearest” mirror (random choice in the country mirrors list), and stores it in a memcached (for up to 1 month). Caching is necessary to avoid switching too often from a mirror to another, which might be serving different versions.
It gets the per-country mirrors lists through an annotated /etc/pacman.d/mirrorlist. For now we have to maintain, if I understood correctly, rsync, website and pacman databases, but are moving to a centralized database (maintained through django), so this should change later.
I didn’t get many opinions for now; feel free to react in comments! I’d also love to hear about other distros policy, and/or get contributions.
P.S.: That’s my first post since I got promoted as a dev in Arch Linux. Thanks to the whole team for letting me in. I don’t have much time these weeks, but should get a lot done next week.
4 Comments to “Distributing packages”
Leave a Reply
Nice if you could make it into the dev-team!! on topic: I would highly appreciate it if there would be any possibility to install an arch system in the future using p2p techniques like bittorrent. maybe it would be possible to establish a central tracker on the archlinux server that distributes the packages, that are afterwards downloaded using aria while installing. i think it wouldn’t be that difficult to generate a .torrent-file of each package.
anyway, i hope you can join the dev group!!
best regards,
kleinlohmi
ps: you have 1 new reader of your rss feed now ;D
OTOH the bittorrent overhead for small files may be too big.
openSUSE has the build service to build and distribute packages. They’re building for other distributions too, so it might be possible to get them to support Arch. :)