Bitcoin uses a simple broadcast network to propagate transactions and blocks. All communications are done over TCP. Bitcoin is fully able to use ports other than 8333 via the -port parameter. IPv6 is supported with Bitcoind/Bitcoin-Qt v0.7.
Detailed in the Protocol documentation.
To connect to a peer, you send a version message containing your version number, block count, and current time. The remote peer will send back a verack message and his own version message if he is accepting connections from your version. You will respond with your own verack if you are accepting connections from his version.
The time data from all of your peers is collected, and the median is used by Bitcoin for all network tasks that use the time (except for other version messages).
You then exchange getaddr and addr messages, storing all addresses that you don't know about. addr messages often contain only one address, but sometimes contain up to 1000. This is most common at the beginning of an exchange.
When someone sends a transaction, they send an inv message containing it to all of their peers. Their peers will request the full transaction with getdata. If they consider the transaction valid after receiving it, they will also broadcast the transaction to all of their peers with an inv, and so on. Peers ask for or relay transactions only if they don't already have them. A peer will never rebroadcast a transaction that it already knows about, though transactions will eventually be forgotten if they don't get into a block after a while. The sender and receiver of the transaction will rebroadcast, however.
Anyone who is generating will collect valid received transactions and work on including them in a block. When someone does find a block, they send an inv containing it to all of their peers, as above. It works the same as transactions.
Everyone broadcasts an addr containing their own IP address every 24 hours. Nodes relay these messages to a couple of their peers and store the address if it's new to them. Through this system, everyone has a reasonably clear picture of which IPs are connected to the network at the moment. After connecting to the network, you get added to everyone's address database almost instantly because of your initial addr.
Network alerts are broadcast with alert messages. No inv-like system is used; these contain the entire alert. If a received alert is valid (signed by one of the people with the private key), it is relayed to all peers. For as long as an alert is still in effect, it is rebroadcast at the start of every new connection.
Initial block download
At the start of a connection, you send a getblocks message containing the hash of the latest block you know about. If the peer doesn't think that this is the latest block, it will send an inv that contains up to 500 blocks ahead of the one you listed. You will then request all of these blocks with getdata, and the peer will send them to you with block messages. After you have downloaded and processed all of these blocks, you will send another getblocks, etc., until you have all of the blocks.
Thin SPV Clients
BIP 0037 introduced support for thin or lite clients by way of Simple Payment Verification. SPV clients do not need to download the full block contents to verify the existence of funds in the blockchain, but rely on the chain of block headers and bloom filters to obtain the data they need from other nodes. This method of client communication allows high security trustless communication with full nodes, but at the expensive of some privacy as the peers can deduce which addresses the SPV client is seeking information about.
MultiBit and bitcoinj as their foundation.work in this fashion using the library
You choose which peers to connect to by sorting your address database by the time since you last saw the address and then adding a bit of randomization.
Bitcoin has three methods of finding peers.
The addr messages described above create an effect similar to the IRC bootstrapping method. You know reasonably quickly whenever a peer joins, though you won't know for a while when they leave.
Bitcoin comes with a list of addresses known as "seed nodes". If you are unable to connect to IRC and you've never connected to the network before, the client will update the address database by connecting to one of the nodes from this list.
The -addnode command line option can be used to manually add a node. The -connect option can force bitcoin to connect only to a specific node.
Bitcoin looks up the IP Addresses of several host names and adds those to the list of potential addresses. This is the default seeding mechanism, as of v0.6.x and later.
As-of version 0.6.x of the Bitcoin client, IRC bootstrapping is no longer enabled by default. The information below is accurate for most versions prior.
Bitcoin joins a random channel between #bitcoin00 and #bitcoin99 on irc.lfnet.org. Your nick is set to an encoded form of your IP address. By decoding all the nicks of all users on the channel, you get a list of all IP addresses currently connected to Bitcoin.
For hosts that cannot make outbound connections on port 6667, the lfnet servers are also listening on port 7777.
If thirty minutes or more has passed since the client has transmitted any messages it will transmit a message to keep the connection to the peer node alive.
If ninety minutes has passed since a peer node has communicated any messages, then the client will assume that connection has closed.