Logsender

The logscraper tool is parsing log files that are available in the directory, attach important data that are provided in buildlog and inventory.yaml files and send it directly to the Opensearch service.

Available arguments for logsender are:

logsender --help

Check log directories and push to the Opensearch service

options:
  -h, --help            show this help message and exit
  --directory DIRECTORY
                        Directory, where the logs will be stored. Defaults to: /tmp/logscraper
  --host HOST           Opensearch host
  --port PORT           Opensearch port
  --username USERNAME   Opensearch username
  --password PASSWORD   Opensearch user password
  --index-prefix INDEX_PREFIX
                        Prefix for the index. Defaults to logstash-
  --index INDEX         Opensearch index. Defaults to: <index-prefix>-YYYY-DD
  --insecure            Skip validating SSL cert
  --follow              Keep sending CI logs
  --workers WORKERS     Worker processes for logsender
  --chunk-size CHUNK_SIZE
                        The bulk chunk size
  --keep                Do not remove log directory after
  --ignore-es-status    Ignore Opensearch bulk
  --debug DEBUG         Be more verbose

Basic usage

Base on the use case, we can run logsender.

Example:

• Send logs to that is running on localhost, skip TLS cert verification

logsender --username logstash --password logstashpassword --host localhost --port 9200 --insecure

• Send logs to service, use 8 workers and ignore Opensearch bulk update status. WARNING: –ignore-es-status should not be used on production environment!

logsender --username logstash --password logstashpassword --host localhost --port 9200 --insecure --workers 8 --ignore-es-status

• Send logs to elasticsaerch service, provide own index name “myindex” and keep log files (they will be not deleted):

logsender --username logstash --password logstashpassword --index myindex --keep

Containerize tool

Instead of using pip tool, you can build your own container image that contains logscraper tool, for example:

docker build -t logscraper -f Dockerfile

NOTE: the logsender tool will be included in logscraper container image.

Then you can execute commands that are described above.

NOTE: The directory where you store log files should be mounted to the container. For example:

podman run \
 --network host \
 -d \
 --name logsender-openstack \
 --volume /mnt/logscraper/openstack:/mnt/logscraper/openstack:z \
 logscraper \
 /usr/local/bin/logsender \
 --username admin \
 --password admin \
 --host localhost \
 --port 9200 \
 --directory /mnt/logscraper/openstack \
 --follow