Low level networking: performance issues

I spent a few hours working on a multicast file transfer program, written in Python, with a friend of mine. The experience was filled with ups and downs. It was good to see that we were able to handle low level stuff such as broadcast packets, for example, in Python. On the other hand, two things were a problem from the start: timing, and raw performance.

Timing problems surfaced out as we were trying to synchronize operations on the reception side. For several reasons, the multicast file transfer can't operate using TCP/IP, or any other unicast flow control algorithm, so we had to implement a low level protocol on top of standard UDP datagrams. Most techniques available to keep the transmission rate under control are rather difficult to implement in this case. In the end, we found out that was very difficult to get optimum performance when trying to receive data and write it to a local file at the same time. Timing was surprisingly critical in this case, and that made the second problem -- raw performance -- specially difficult to live with. We tried to use threads, and several other techniques, with little or no luck. We either have too long queues, or a low effective transfer rate. We knew it was going to be hard, but this is going beyond our worst-case plans.

Our best bet, at this point, is that a clever rate control algorithm can be used to alleviate some of the problems we had so far; we intend to transmit data at just the right speed, using a rate that is fast enough to give good performance, but still not so much as not to fill reception buffers to the point where we drop packets. We're going to try it over the next week. We hope to have good results to present in a very short time.