# Fluentd BigQuery Sample

Fluentd BigQuery Sample is an archived Google Docker sample, no longer maintained, that pairs nginx with Fluentd to stream web-server access logs into Google BigQuery for experimenting with log collection. It is open source and self-hosted, installable on Docker Hub.

Canonical page: https://ourlocalhost.com/app/5179/fluentd-bigquery-sample

## Description

Fluentd BigQuery Sample is an archived Google Docker sample, no longer maintained, that pairs nginx with Fluentd to stream web-server access logs into Google BigQuery for experimenting with log collection.

## Platform availability

Available on 0 of 10 platforms.

## Health score

32/100 (Low), based on 5 of 5 factors.
- Maintained: 0 (GitHub repository is archived)
- Popular: 31 (35 GitHub stars)
- Easy to install: 25 (0 install platforms, install notes, Docker image)
- Light to run: 100 (30 MB minimum RAM)
- Security check: 54 (no serious findings, OpenSSF Scorecard 2.2/10)

## Links

- Docker Hub: https://hub.docker.com/r/google/fluentd-bigquery-sample
- Documentation: https://cloud.google.com/solutions/real-time/fluentd-bigquery
- GitHub: https://github.com/GoogleCloudPlatform/bigquery-fluentd-docker-sample

## Common questions

### How do I install and start the sample?

Run bundle install, then bundle exec fluentd -c fluentd.conf. This installs the add-on packages Fluentd needs and starts it with the example config.

### Which port must be reachable for another host to send logs to Fluentd?

Check the startup log for a line like listening fluent socket on 0.0.0.0:24224. If another host is sending logs, that port must be reachable.

### Does the optional Fluentd UI require opening a port?

You can install the fluentd-ui gem and run fluentd-ui start to get a simple web page for checking Fluentd. It listens on port 9292, so open that port only if you want to use the UI.

### What thread and flush settings are recommended for inserts over the internet?

For inserts over the internet, 10 or more threads seems good, while fewer threads may be better for Google Compute Engine instances with low latency to BigQuery. The buffer/flush_interval setting controls the interval between data flushes, with a default of 0.25 and subsecond values such as 0.15 allowed on Fluentd v0.10.42 or later.

### How does the Docker sample get web server access logs into BigQuery?

It launches a Docker container that is preconfigured with nginx and Fluentd. It then uses a Fluentd-to-BigQuery plugin to load web server access logs into Google BigQuery in near-real-time.

## Alternatives

- Datadog: Datadog Agent is an on-host monitoring agent that collects metrics, logs, and traces from hosts, containers, and Kubernetes clusters and se…
- Fluentd: Fluentd is an open-source log collector that gathers events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, H…
- Grafana Loki: Grafana Loki is an open-source log aggregation system.
- Graylog: Graylog is a centralized log management platform built on open standards for capturing, storing, and analyzing terabytes of machine data in…
- Kibana: Kibana is the data visualization layer for querying, analyzing, and managing data stored in Elasticsearch.
- Logstash: Logstash is a server-side data processing pipeline that ingests data from many sources, transforms it, and forwards it to a destination suc…
- Splunk: Splunk is a self-hosted platform for collecting, indexing, searching, and analyzing log and security data, helping homelabbers and operatio…
- telegraf: Telegraf is a standalone, statically compiled agent for collecting, processing, aggregating, and writing system metrics, logs, and arbitrar…

---
Source: Our Local Host (https://ourlocalhost.com/app/5179/fluentd-bigquery-sample), the self-hosted and open-source app directory.
