Decentralized Downloads for Bazel: Introducing bazel_rules_bt

Find the project page at https://github.com/filmil/bazel_rules_bt

In the world of reproducible builds and efficient dependency management, Bazel stands out as a powerful tool. However, fetching large or uncommon dependencies can sometimes be a bottleneck. Enter bazel_rules_bt, a set of Bazel rules that uses the BitTorrent protocol to bring decentralized file downloading capabilities directly into your build system.

The Power of Peer-to-Peer for Builds

bazel_rules_bt addresses a common challenge: efficiently acquiring external files and archives that might be hosted on less reliable servers, or when you want to take advantage of peer-to-peer distribution for speed and resilience. By integrating the BitTorrent protocol, this extension offers a robust and potentially faster alternative to traditional HTTP/S downloads, especially for widely available public data or large files.

The rules are built upon rules_multitool and its embedded rain torrent client, ensuring a streamlined and self-contained solution for Bazel users.

Getting Started with bazel_rules_bt

Integrating bazel_rules_bt into your Bazel project is straightforward.

Setup

To begin, add the following dependency to your MODULE.bazel file:

bazel_dep(name = "bazel_rules_bt", version = "0.0.0") # Replace with the actual version

Next, load the necessary rules in your repo.bzl file:

load("@bazel_rules_bt//:repo.bzl", "bt_file")

Usage

bazel_rules_bt provides two primary rules: bt_file for single file downloads and bt_archive for downloading and extracting archives.

bt_file: Downloading a Single File

The bt_file rule allows you to download a specific file from a torrent. This is ideal for obtaining individual binary dependencies, configuration files, or any single asset.

Here’s how to use bt_file in your MODULE.bazel:

load("@bazel_rules_bt//:repo.bzl", "bt_file")
bt_file(
    name = "my_file",
    uri = "magnet:?xt=urn:btih:...", # Or a URL to a .torrent file
)

Once defined, the downloaded file will be accessible within your Bazel workspace at @my_file//:file.

Attributes for bt_file:

bt_archive: Downloading and Extracting Archives

For more complex dependencies distributed as archives (e.g., zip, tar, gzip), the bt_archive rule automates the download, extraction, and integration into your Bazel workspace.

Here’s an example in your MODULE.bazel:

load("@bazel_rules_bt//:repo.bzl", "bt_archive")
bt_archive(
    name = "my_archive",
    uri = "magnet:?xt=urn:btih:...", # Or a URL to a .torrent file
    strip_prefix = "my_archive-1.0",
    build_file_content = """
filegroup(
    name = "my_archive_files",
    srcs = glob(["**"]),
    visibility = ["//visibility:public"],
)
""",
)

This configuration will download the archive, extract its contents, and make them available as @my_archive//:my_archive_files. The strip_prefix attribute is particularly useful for handling common archive structures where the contents are nested within a top-level directory. The build_file_content allows you to define a BUILD file directly within the rule, enabling Bazel to correctly interpret and use the extracted files.

Attributes for bt_archive:

Conclusion

bazel_rules_bt offers a solution for integrating BitTorrent-based file acquisition into your Bazel build workflows. By leveraging the decentralized nature of BitTorrent, it provides a potentially more resilient way to manage external dependencies, especially for large datasets or scenarios where traditional download methods might fall short.