Fluentd BigQuery Sample

Fluentd BigQuery Sample is an archived Google Docker sample, no longer maintained, that pairs nginx with Fluentd to stream web-server access logs into Google…

  • Databases
  • Monitoring & Logging
  • Tools & Utilities
Fluentd BigQuery Sample is an archived Google Docker sample, no longer maintained, that pairs nginx with Fluentd to stream web-server access logs into Google BigQuery for experimenting with log collection.

Platform availability

Available on 0 of 10 platforms

  • Unraid (not listed)
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

32/100 Low

Maintained
0GitHub repository is archived
Popular
3135 GitHub stars
Easy to install
250 install platforms, install notes, Docker image
Light to run
10030 MB minimum RAM
Security check
54no serious findings, OpenSSF Scorecard 2.2/10
GitHub stars
35
Open issues
0
Last commit
2019-04-23
License
Apache-2.0
Activity
stale

Checked 15 days ago - source: GitHub (GoogleCloudPlatform/bigquery-fluentd-docker-sample)

Resources & Compatibility

Minimum requirements: 30 MB RAM. ARM (e.g. Raspberry Pi) is not supported.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Install gems, then start the sample Run bundle install, then bundle exec fluentd -c fluentd.conf. This installs the add-on packages Fluentd needs and starts it with the example config, so the sample is ready to process logs. source
  • Check the startup log for the port When Fluentd starts, look for a line like listening fluent socket on 0.0.0.0:24224. That tells you it is ready to receive forwarded logs; if another host is sending logs, that port must be reachable. source
  • Optional: start Fluentd UI on port 9292 You can install the fluentd-ui gem and run fluentd-ui start to get a simple web page for checking Fluentd. It listens on port 9292, so open that port only if you want to use the UI. source
  • Status Upstream marks this as an outdated sample and points to Google's updated Fluentd and BigQuery content; treat it as a legacy reference setup. source
  • Note This sample is outdated; follow the updated Fluentd-to-BigQuery tutorial for current steps. source
  • Docker setup Run the Docker image, which pre-configures nginx and Fluentd; the sample is outdated, so use the linked documentation for current steps. source

Fetched 7 days ago - each note links to its source

Alternatives (8)

  • Datadog - Datadog Agent is an on-host monitoring agent that collects metrics, logs, and traces from hosts, containers, and Kubernetes clusters and se…
  • Fluentd - Fluentd is an open-source log collector that gathers events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, H…
  • Grafana Loki - Grafana Loki is an open-source log aggregation system.
  • Graylog - Graylog is a centralized log management platform built on open standards for capturing, storing, and analyzing terabytes of machine data in…
  • Kibana - Kibana is the data visualization layer for querying, analyzing, and managing data stored in Elasticsearch.
  • Logstash - Logstash is a server-side data processing pipeline that ingests data from many sources, transforms it, and forwards it to a destination suc…
  • Splunk - Splunk is a self-hosted platform for collecting, indexing, searching, and analyzing log and security data, helping homelabbers and operatio…
  • telegraf - Telegraf is a standalone, statically compiled agent for collecting, processing, aggregating, and writing system metrics, logs, and arbitrar…

Checked 40 days ago - source: AlternativeTo

Common questions

How do I install and start the sample?

Run bundle install, then bundle exec fluentd -c fluentd.conf. This installs the add-on packages Fluentd needs and starts it with the example config.

Which port must be reachable for another host to send logs to Fluentd?

Check the startup log for a line like listening fluent socket on 0.0.0.0:24224. If another host is sending logs, that port must be reachable.

Does the optional Fluentd UI require opening a port?

You can install the fluentd-ui gem and run fluentd-ui start to get a simple web page for checking Fluentd. It listens on port 9292, so open that port only if you want to use the UI.

What thread and flush settings are recommended for inserts over the internet?

For inserts over the internet, 10 or more threads seems good, while fewer threads may be better for Google Compute Engine instances with low latency to BigQuery. The buffer/flush_interval setting controls the interval between data flushes, with a default of 0.25 and subsecond values such as 0.15 allowed on Fluentd v0.10.42 or later.

How does the Docker sample get web server access logs into BigQuery?

It launches a Docker container that is preconfigured with nginx and Fluentd. It then uses a Fluentd-to-BigQuery plugin to load web server access logs into Google BigQuery in near-real-time.

Answers sourced from docs.fluentd.org, docs.fluentd.org, docs.fluentd.org, github.com, github.com

Community

Collected 48 days ago - public community threads