Here's a list of the BR variables, which configure the mirroring behavior:
BR2_PRIMARY_SITE BR2_PRIMARY_SITE_ONLY BR2_BACKUP_SITE BR2_KERNEL_MIRROR BR2_GNU_MIRROR BR2_LUAROCKS_MIRROR BR2_CPAN_MIRROR
The variables usage should be pretty obvious, but as always don't hesitate to take a look at the Config.in, where the variables are documented (or use menuconfig and go to "Build options", "Mirrors and Download locations").
Basically you have 2 ways to change these variables - either edit them in Config.in file, or assign them in your boards's configuration file (configs/boardname_defconfig). The difference is that boardname_defconfig affects only your board, and Config.in affects all boards' builds.
Here's a very simple example of making a local mirror for all sources, just add this to your boardname_defconfig:
# Use local source mirror BR2_PRIMARY_SITE="http://www.myserver.org/mirrors/buildroot"
Of course the best place to put your server is inside your LAN, which will guarantee sub-second downloads during the build process.