I have one requirement to run a logstash pipeline which should send the logs to Google storage cloud. It should rotate the log for every one hour and send to GCP bucket.Also my main requirement is, the logs should save in a folder and date folder should dynamically create.
AWS s3 outputs are working fine for me, need the configuarion for GCP bucket. Anyone can you please share your logstash configuration file.
For testing purpose, I tried by using file input and output with 30 seconds. Please correct me if I am wrong.
input {
file {
path => "/tmp/*.log"
type => "log"
}
}
output {
google_cloud_storage {
bucket => "gcp-bucket/log_folder/%Y-%m-%d"
date_pattern => "%Y-%m-%dT%S:30"
uploader_interval_secs => 30
log_file_prefix => "test"
gzip_content_encoding => true
gzip => true
include_hostname => true
include_uuid => true
output_format => json
temp_directory => "/tmp/logstash-gcs"
}
}