# Native Extensions: Precompiled rdkafka-ruby Karafka uses the `rdkafka-ruby` gem, which includes a native C extension that wraps the librdkafka library. To provide faster and more reliable installation, we distribute **native extensions** as precompiled gems (also called "native gems") for major platforms. This eliminates the need to compile C extensions during installation, resulting in significantly faster gem installation and removing build dependency requirements. ## Native Extensions: Precompiled rdkafka-ruby / What Are Native Extensions? Native extensions are platform-specific compiled binaries that contain: - **Pre-compiled librdkafka libraries** with all dependencies statically linked - **Self-contained binaries** that don't require system dependencies - **Cryptographically verified dependencies** with SHA256 checksums for supply chain security - **Full feature support** including SSL/TLS, SASL, Kerberos/GSSAPI, and compression ## Native Extensions: Precompiled rdkafka-ruby / Supported Platforms `rdkafka-ruby` ships native extensions for the following platforms:
Platform Architecture Minimum Version glibc/musl Compatible Distributions
Linux (glibc) x86_64 Ubuntu 22.04 LTS glibc 2.35+ Ubuntu 22.04+, CentOS 9+, RHEL 9+, Debian 12+, Fedora 36+
Linux (musl) x86_64 Alpine 3.18+ musl 1.2.4+ Alpine 3.18+, and other musl-based distributions
macOS arm64 macOS 14+ N/A Apple Silicon Macs (M1, M2, M3+)
### Native Extensions: Precompiled rdkafka-ruby / Supported Platforms / Checking Platform Support To check if your platform is supported: ```shell # Check your platform ruby -e 'puts Gem::Platform.local.to_s' ``` **Expected output examples:** - `x86_64-linux-gnu` (Ubuntu/CentOS/RHEL) - `x86_64-linux-musl` (Alpine Linux) - `arm64-darwin` (Apple Silicon Mac) ### Native Extensions: Precompiled rdkafka-ruby / Supported Platforms / Checking Library Versions **For glibc-based systems:** ```shell # Check glibc version ldd --version ``` **For musl-based systems:** ```shell # Check musl version ldd 2>&1 | head -1 ``` If your platform isn't supported or doesn't meet the minimum requirements, `rdkafka-ruby` will automatically fall back to source compilation. ## Native Extensions: Precompiled rdkafka-ruby / Installation Benefits **Benefits:** - **10-100x faster installation** (seconds instead of minutes) - **More reliable** - no compilation failures - **No build dependencies required** - works in minimal containers - **Enhanced security** - all dependencies cryptographically verified - **Cloud-ready** - perfect for Docker, AWS Lambda, etc. ### Native Extensions: Precompiled rdkafka-ruby / Installation Benefits / Without Native Extensions (Fallback) ```shell $ gem install rdkafka --platform=ruby # Downloads source gem and compiles during installation # Requires: build tools, librdkafka, OpenSSL, SASL, Kerberos, etc. ``` ## Native Extensions: Precompiled rdkafka-ruby / What's Included in Native Extensions Each native extension includes a self-contained librdkafka library with: ### Native Extensions: Precompiled rdkafka-ruby / What's Included in Native Extensions / Core Dependencies (Statically Linked) - **librdkafka** - The core Kafka client library - **OpenSSL** - SSL/TLS encryption support - **Cyrus SASL** - Authentication mechanisms (PLAIN, SCRAM, GSSAPI) - **MIT Kerberos** - Kerberos/GSSAPI authentication for enterprise environments - **zlib** - Standard compression - **ZStd** - High-performance compression ## Native Extensions: Precompiled rdkafka-ruby / Build Process and Security ### Native Extensions: Precompiled rdkafka-ruby / Build Process and Security / Supply Chain Security All dependencies are verified with SHA256 checksums during the build process: ```shell # Example from build process [SECURITY] Verifying checksum for openssl-3.0.16.tar.gz... [SECURITY] ✅ Checksum verified for openssl-3.0.16.tar.gz [SECURITY] 🔒 SECURITY VERIFICATION COMPLETE [SECURITY] All dependencies downloaded and verified with SHA256 checksums ``` ### Native Extensions: Precompiled rdkafka-ruby / Build Process and Security / Automated Build Pipeline Native extensions are built using GitHub Actions with: 1. **Dependency Download**: All dependencies downloaded from official sources 2. **Checksum Verification**: SHA256 verification for supply chain security 3. **Static Compilation**: All dependencies statically linked 4. **Self-Contained Packaging**: No external dependencies required 5. **Automated Testing**: Comprehensive test suite across Ruby versions 6. **Trusted Publishing**: Cryptographic attestation via RubyGems ## Native Extensions: Precompiled rdkafka-ruby / Docker and Container Usage Native extensions are ideal for containerized applications: ```dockerfile FROM ruby:3.4-slim # No build dependencies needed! RUN gem install karafka COPY . /app WORKDIR /app ``` ### Native Extensions: Precompiled rdkafka-ruby / Docker and Container Usage / Before (with compilation) ```dockerfile FROM ruby:3.4 RUN apt-get update && apt-get install -y \ build-essential \ librdkafka-dev \ libsasl2-dev \ libssl-dev RUN gem install karafka # 2-15 minutes ``` ### Native Extensions: Precompiled rdkafka-ruby / Docker and Container Usage / After (with native extensions) ```dockerfile FROM ruby:3.4-slim RUN gem install karafka # 3-10 seconds ``` ## Native Extensions: Precompiled rdkafka-ruby / Troubleshooting ### Native Extensions: Precompiled rdkafka-ruby / Troubleshooting / Force Native Extension Installation ```shell # Explicitly request native extension gem install rdkafka --platform=x86_64-linux-gnu # Or in Gemfile gem 'rdkafka', platforms: [:x86_64_linux_gnu] ``` ### Native Extensions: Precompiled rdkafka-ruby / Troubleshooting / Fallback to Source Compilation If native extensions don't work for your platform: ```shell # Force source compilation gem install rdkafka --platform=ruby # Or in Gemfile gem 'rdkafka', force_ruby_platform: true ``` ## Native Extensions: Precompiled rdkafka-ruby / Source Compilation (Fallback) If you need to use source compilation instead of native extensions (e.g., for custom configurations or unsupported platforms): ### Native Extensions: Precompiled rdkafka-ruby / Source Compilation (Fallback) / Force Source Compilation ```shell # Force source compilation gem install rdkafka --platform=ruby # Or in Gemfile gem 'rdkafka', force_ruby_platform: true ``` ## Native Extensions: Precompiled rdkafka-ruby / Migration from Source Compilation If you're currently using source compilation: 1. **Remove build dependencies** from your Dockerfile/CI 2. **Update Gemfile** to allow native extensions: ```ruby # Remove platform restrictions gem 'rdkafka' # Will automatically use native extensions ``` 3. **Rebuild** your containers - they'll be much faster! --- *Last modified: 2025-09-01 12:11:21*