Fluentd BigQuery Sample
Fluentd BigQuery Sample is an archived Google Docker sample, no longer maintained, that pairs nginx with Fluentd to stream web-server access logs into Google…
Platform availability
Available on 0 of 10 platforms
- Unraid (not listed)
- TrueNAS (not listed)
- Umbrel (not listed)
- ZimaOS (not listed)
- Proxmox (not listed)
- Helm (not listed)
- Mac (not listed)
- Windows (not listed)
- Linux (not listed)
- Any server (not listed)
Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.
32/100 Low
- Maintained
- 0GitHub repository is archived
- Popular
- 3135 GitHub stars
- Easy to install
- 250 install platforms, install notes, Docker image
- Light to run
- 10030 MB minimum RAM
- Security check
- 54no serious findings, OpenSSF Scorecard 2.2/10
- GitHub stars
- 35
- Open issues
- 0
- Last commit
- 2019-04-23
- License
- Apache-2.0
- Activity
- stale
Checked 15 days ago - source: GitHub (GoogleCloudPlatform/bigquery-fluentd-docker-sample)
Resources & Compatibility
Minimum requirements: 30 MB RAM. ARM (e.g. Raspberry Pi) is not supported.
Checked 54 days ago - source: Docker Hub image tags and published docs
First-install notes
- Install gems, then start the sample Run bundle install, then bundle exec fluentd -c fluentd.conf. This installs the add-on packages Fluentd needs and starts it with the example config, so the sample is ready to process logs. source
- Check the startup log for the port When Fluentd starts, look for a line like listening fluent socket on 0.0.0.0:24224. That tells you it is ready to receive forwarded logs; if another host is sending logs, that port must be reachable. source
- Optional: start Fluentd UI on port 9292 You can install the fluentd-ui gem and run fluentd-ui start to get a simple web page for checking Fluentd. It listens on port 9292, so open that port only if you want to use the UI. source
- Status Upstream marks this as an outdated sample and points to Google's updated Fluentd and BigQuery content; treat it as a legacy reference setup. source
- Note This sample is outdated; follow the updated Fluentd-to-BigQuery tutorial for current steps. source
- Docker setup Run the Docker image, which pre-configures nginx and Fluentd; the sample is outdated, so use the linked documentation for current steps. source
Fetched 7 days ago - each note links to its source
Variants
Alternatives (8)
- Datadog - Datadog Agent is an on-host monitoring agent that collects metrics, logs, and traces from hosts, containers, and Kubernetes clusters and se…
- Fluentd - Fluentd is an open-source log collector that gathers events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, H…
- Grafana Loki - Grafana Loki is an open-source log aggregation system.
- Graylog - Graylog is a centralized log management platform built on open standards for capturing, storing, and analyzing terabytes of machine data in…
- Kibana - Kibana is the data visualization layer for querying, analyzing, and managing data stored in Elasticsearch.
- Logstash - Logstash is a server-side data processing pipeline that ingests data from many sources, transforms it, and forwards it to a destination suc…
- Splunk - Splunk is a self-hosted platform for collecting, indexing, searching, and analyzing log and security data, helping homelabbers and operatio…
- telegraf - Telegraf is a standalone, statically compiled agent for collecting, processing, aggregating, and writing system metrics, logs, and arbitrar…
Checked 40 days ago - source: AlternativeTo
Common questions
How do I install and start the sample?
Run bundle install, then bundle exec fluentd -c fluentd.conf. This installs the add-on packages Fluentd needs and starts it with the example config.
Which port must be reachable for another host to send logs to Fluentd?
Check the startup log for a line like listening fluent socket on 0.0.0.0:24224. If another host is sending logs, that port must be reachable.
Does the optional Fluentd UI require opening a port?
You can install the fluentd-ui gem and run fluentd-ui start to get a simple web page for checking Fluentd. It listens on port 9292, so open that port only if you want to use the UI.
What thread and flush settings are recommended for inserts over the internet?
For inserts over the internet, 10 or more threads seems good, while fewer threads may be better for Google Compute Engine instances with low latency to BigQuery. The buffer/flush_interval setting controls the interval between data flushes, with a default of 0.25 and subsecond values such as 0.15 allowed on Fluentd v0.10.42 or later.
How does the Docker sample get web server access logs into BigQuery?
It launches a Docker container that is preconfigured with nginx and Fluentd. It then uses a Fluentd-to-BigQuery plugin to load web server access logs into Google BigQuery in near-real-time.
Answers sourced from docs.fluentd.org, docs.fluentd.org, docs.fluentd.org, github.com, github.com
Community
Collected 48 days ago - public community threads