The connective tissue framework and sheath of a nerve which bind together the nerve bundles, each of which has its own special sheath, or perineurium.