awss3exporter

package module

v0.130.0 Latest Latest Go to latest Published: Jul 14, 2025 License: Apache-2.0 Imports: 29 Imported by: 6

README ¶

AWS S3 Exporter for OpenTelemetry Collector

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code coverage
Code Owners	@atoulme, @pdelewski, @Erog38

Schema supported

This exporter targets to support proto/json format.

Exporter Configuration

The following exporter configuration parameters are supported.

Name	Description	Default
`region`	AWS region.	"us-east-1"
`s3_bucket`	S3 bucket
`s3_prefix`	prefix for the S3 key (root directory inside bucket).
`s3_partition_format`	filepath formatting for the partition; See strftime for format specification.	"year=%Y/month=%m/day=%d/hour=%H/minute=%M"
`role_arn`	the Role ARN to be assumed
`file_prefix`	file prefix defined by user
`marshaler`	marshaler used to produce output data	`otlp_json`
`encoding`	Encoding extension to use to marshal data. Overrides the `marshaler` configuration option if set.
`encoding_file_extension`	file format extension suffix when using the `encoding` configuration option. May be left empty for no suffix to be appended.
`endpoint`	(REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket`
`storage_class`	S3 storageclass	STANDARD
`acl`	S3 Object Canned ACL	none (does not set by default)
`s3_force_path_style`	set this to `true` to force the request to use path-style addressing	false
`disable_ssl`	set this to `true` to disable SSL when sending requests	false
`compression`	should the file be compressed	none
`sending_queue`	exporters common queuing	disabled
`timeout`	exporters common timeout	5s
`resource_attrs_to_s3`	determines the mapping of S3 configuration values to resource attribute values for uploading operations.
`retry_mode`	The retryer implementation, the supported values are "standard", "adaptive" and "nop". "nop" will set the retryer as `aws.NopRetryer`, which effectively disable the retry.	standard
`retry_max_attempts`	The max number of attempts for retrying a request if the `retry_mode` is set. Setting max attempts to 0 will allow the SDK to retry all retryable errors until the request succeeds, or a non-retryable error is returned.	3
`retry_max_backoff`	the max backoff delay that can occur before retrying a request if `retry_mode` is set	20s

Marshaler

Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:

otlp_json (default): the OpenTelemetry Protocol format, represented as json.
otlp_proto: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.
sumo_ic: the Sumo Logic Installed Collector Archive format. This format is supported only for logs.
body: export the log body as string. This format is supported only for logs.

Encoding

Encoding overrides marshaler if present and sets to use an encoding extension defined in the collector configuration.

See https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding.

Compression

none (default): No compression will be applied
gzip: Files will be compressed with gzip. This does not support sumo_icmarshaler.

resource_attrs_to_s3

s3_prefix: Defines which resource attribute's value should be used as the S3 prefix. When this option is set, it dynamically overrides s3uploader/s3_prefix. If the specified resource attribute exists in the data,
its value will be used as the prefix; otherwise, s3uploader/s3_prefix will serve as the fallback.

Example Configurations

Following example configuration defines to store output in 'eu-central' region and bucket named 'databucket'.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'

    # Optional (disabled by default)
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 100

    # Optional (5s by default)
    timeout: 20s

Logs and traces will be stored inside 'databucket' in the following path format.

metric/year=YYYY/month=MM/day=DD/hour=HH/minute=mm

Partition Formatting

By setting the s3_partition_format option, users can specify the file path for their logs. See the strftime reference for more formatting options.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'

In this case, logs and traces would be stored in the following path format.

metric/YYYY/MM/DD/HH/mm

Data routing based on resource attributes

When resource_attrs_to_s3/s3_prefix is configured, the S3 prefix is dynamically derived from a specified resource attribute in your data. If the attribute value is unavailable, the prefix will fall back to the value defined in s3uploader/s3_prefix.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'
    resource_attrs_to_s3:
      s3_prefix: "com.awss3.prefix"

In this case, metrics, logs and traces would be stored in the following path format examples:

prefix1/YYYY/MM/DD/HH/mm
foo-prefix/YYYY/MM/DD/HH/mm
prefix-bar/YYYY/MM/DD/HH/mm
metric/YYYY/MM/DD/HH/mm
...

Retry

Standard is the default retryer implementation used by service clients. See the retry package documentation for details on what errors are considered as retryable by the standard retryer implementation.

See also the aws-sdk-go reference for more information.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      retry_mode: "standard"
      retry_max_attempts: 5
      retry_max_backoff: "30s"

AWS Credential Configuration

This exporter follows default credential resolution for the aws-sdk-go.

Follow the guidelines for the credential configuration.

OpenTelemetry Collector Helm Chart for Kubernetes

For example, when using OpenTelemetry Collector Helm Chart you could use extraEnvs in the values.yaml.

extraEnvs:
- name: AWS_ACCESS_KEY_ID
  value: "< YOUR AWS ACCESS KEY >"
- name: AWS_SECRET_ACCESS_KEY
  value: "< YOUR AWS SECRET ACCESS KEY >"

Documentation ¶

Overview ¶

Package awss3exporter stores OpenTelemetry data as an AWS S3 exporter.

Constants ¶

View Source

const (
	DefaultRetryMode        = "standard"
	DefaultRetryMaxAttempts = 3
	DefaultRetryMaxBackoff  = 20 * time.Second
)

View Source

const (
	SourceCategoryKey = "_sourceCategory"
	SourceHostKey     = "_sourceHost"
	SourceNameKey     = "_sourceName"
)

Variables ¶

View Source

var ErrUnknownMarshaler = errors.New("unknown marshaler")

Functions ¶

func NewFactory ¶

func NewFactory() exporter.Factory

NewFactory creates a factory for S3 exporter.

Types ¶

type Config ¶

type Config struct {
	QueueSettings   exporterhelper.QueueBatchConfig `mapstructure:"sending_queue"`
	TimeoutSettings exporterhelper.TimeoutConfig    `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
	S3Uploader      S3UploaderConfig                `mapstructure:"s3uploader"`
	MarshalerName   MarshalerType                   `mapstructure:"marshaler"`

	// Encoding to apply. If present, overrides the marshaler configuration option.
	Encoding              *component.ID     `mapstructure:"encoding"`
	EncodingFileExtension string            `mapstructure:"encoding_file_extension"`
	ResourceAttrsToS3     ResourceAttrsToS3 `mapstructure:"resource_attrs_to_s3"`
}

Config contains the main configuration options for the s3 exporter

func (*Config) Validate ¶ added in v0.83.0

func (c *Config) Validate() error

type MarshalerType ¶ added in v0.76.1

type MarshalerType string

const (
	OtlpProtobuf MarshalerType = "otlp_proto"
	OtlpJSON     MarshalerType = "otlp_json"
	SumoIC       MarshalerType = "sumo_ic"
	Body         MarshalerType = "body"
)

type ResourceAttrsToS3 ¶ added in v0.125.0

type ResourceAttrsToS3 struct {
	// S3Prefix indicates the mapping of the key (directory) prefix used for writing into the bucket to a specific resource attribute value.
	S3Prefix string `mapstructure:"s3_prefix"`
	// contains filtered or unexported fields
}

ResourceAttrsToS3 defines the mapping of S3 uploading configuration values to resource attribute values.

type S3UploaderConfig ¶

type S3UploaderConfig struct {
	Region string `mapstructure:"region"`
	// S3Bucket is the bucket name to be uploaded to.
	S3Bucket string `mapstructure:"s3_bucket"`
	// S3Prefix is the key (directory) prefix to written to inside the bucket
	S3Prefix string `mapstructure:"s3_prefix"`
	// S3PartitionFormat is used to provide the rollup on how data is written. Uses [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) formatting.
	S3PartitionFormat string `mapstructure:"s3_partition_format"`
	// FilePrefix is the filename prefix used for the file to avoid any potential collisions.
	FilePrefix string `mapstructure:"file_prefix"`
	// Endpoint is the URL used for communicated with S3.
	Endpoint string `mapstructure:"endpoint"`
	// RoleArn is the role policy to use when interacting with S3
	RoleArn string `mapstructure:"role_arn"`
	// S3ForcePathStyle sets the value for force path style.
	S3ForcePathStyle bool `mapstructure:"s3_force_path_style"`
	// DisableSLL forces communication to happen via HTTP instead of HTTPS.
	DisableSSL bool `mapstructure:"disable_ssl"`
	// ACL is the canned ACL to use when uploading objects.
	ACL string `mapstructure:"acl"`

	StorageClass string `mapstructure:"storage_class"`
	// Compression sets the algorithm used to process the payload
	// before uploading to S3.
	// Valid values are: `gzip` or no value set.
	Compression configcompression.Type `mapstructure:"compression"`

	// RetryMode specifies the retry mode for S3 client, default is "standard".
	// Valid values are: "standard", "adaptive", or "nop".
	// "nop" will disable retry by setting the retryer to aws.NopRetryer.
	RetryMode string `mapstructure:"retry_mode"`
	// RetryMaxAttempts specifies the maximum number of attempts for S3 client.
	// Default is 3 (SDK default).
	RetryMaxAttempts int `mapstructure:"retry_max_attempts"`
	// RetryMaxBackoff specifies the maximum backoff delay for S3 client.
	// Default is 20 seconds (SDK default).
	RetryMaxBackoff time.Duration `mapstructure:"retry_max_backoff"`
}

S3UploaderConfig contains aws s3 uploader related config to controls things like bucket, prefix, batching, connections, retries, etc.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
internal
metadata
upload

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL