Get started
Yandex Object Storage

Logging actions with a bucket

Written by

In Object Storage, there is an option to log all actions with a bucket. You can record logs, for example, to run an internal security audit or get more granular information about bucket operations.

Logging is disabled by default. After you enable this option, Object Storage will write actions with the bucket to an object once an hour.

To save logs, do the following:

  • Define the source bucket that you want to log actions with.
  • Create a target bucket where you want to save the logs.
  • Enable logging.
  • (optional) Select the prefix of the object key.

Prerequisites

The source and target buckets must be in the same cloud.

Format of the key for the log object

Object Storage uses the following format of the key for the log object:

<prefix>/YYYY-MM-DD-HH-MM-SS-<ID>

Where:

  • prefix/: The prefix of the key for the log object. You can specify your own prefix when enabling logging.
  • YYYY-MM-DD-HH-MM-SS: Date and time of saving the log object in the target bucket (UTC format).
  • ID: A unique record ID that prevents the object from being overwritten.

Prefix of the key

The key prefix lets you distinguish:

  • Data belonging to different buckets, if the logs for multiple source buckets are saved to the same target bucket.
  • Logging actions from other actions with the bucket, if the logs are saved to the source bucket. That's because the logging operation is also considered an action with the bucket in this case.
  • The log objects from other objects, in order to delete logs from time to time. You can set up the lifecycle for the target bucket so that objects with a specific key prefix are deleted automatically.

Format of the log object

Logs are saved to a JSON file. For every action with the bucket, a record is written to the file in the following format:

Field Type Description
bucket String Bucket name.
bytes_received Int64 Size of the request in bytes.
bytes_send Int64 Response size in bytes.
handler String Request method in the REST format.<HTTP method>.<subject>.
http_referer String URL of the request source.
ip String User's IP address.
method String HTTP request method.
object_key String The object's key in URL encoded format.
protocol String Data transfer protocol version.
range String An HTTP header that defines the range of bytes to load from the object.
requester String User ID.
request_args String Arguments of the URL request.
request_id String Request ID.
request_path String Full path of the request.
request_time Int64 Request processing time, in milliseconds.
scheme String Type of data transfer protocol.
Acceptable values:
- http, an application layer protocol.
- https, an application layer protocol with encryption support.
ssl_protocol String Security protocol.
status Int64 HTTP response code.
storage_class String Storage class of the object.
timestamp String Date and time of the operation with the bucket, in the YYYY-MM-DDTHH:MM:MMZ format.
user_agent String Client application (User Agent) that executed the request.
version_id String Version of the object.
vhost String Virtual host of the request.
Acceptable values:
– storage.yandexcloud.net.
– bucket name>.storage.yandexcloud.net.
– website.yandexcloud.net.
– <bucket name>.website.yandexcloud.net.

Example of a record in the log file:

{
  "bucket": "my-bucket-example",
  "bytes_received": 749,
  "bytes_send": 1251,
  "handler": "REST.GET.OBJECT",
  "http_referer": "https://example.com/page",
  "ip": "84.201.121.46",
  "method": "GET",
  "object_key": "path/logs/2020-11-10-14-42-11-123f57b5-1853-4120-8d7a-5bcc1e9e9b4f",
  "protocol": "HTTP/1.1",
  "range": "-",
  "requester": "-",
  "request_args": "X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Date=20201030T072100Z\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Expires=43200\u0026X-Amz-Credential=ZGB4EY1...\u0026X-Amz-Signature=12f350...",
  "request_id": "1235efdab7445028",
  "request_path": "/my-bucket-example/path/logs/2020-11-10-14-42-11-123f57b5-1853-4120-8d7a-5bcc1e9e9b4f?X-Amz-...",
  "request_time":88,
  "scheme": "https",
  "ssl_protocol": "TLSv1.2",
  "status": 200,
  "storage_class": "STANDARD",
  "timestamp": "2020-11-10T13:21:18Z",
  "user_agent": "docker/19.03.9 go/go1.13.10 git-commit/1d238398e7 kernel/4.4.0-142-generic os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)",
  "version_id": "",
  "vhost": "storage.yandexcloud.net"
}

Logging specifics

There are several points to note about how actions with a bucket are logged in Object Storage.

Best-effort log delivery

Most requests to a bucket are written to the log file (if the bucket was set up correctly to support logging). Most records are written within a few hours after the request is actually processed.

However, Object Storage doesn't guarantee that the logs are saved in a complete and timely manner. It may take several hours to record an action with the bucket in a log file. In some cases, a record might fail to appear in the file.

The log file provides an overview of the nature of traffic in the bucket, but is not intended for logging every request. In the payment documents, you can find several requests that are not saved in the log file.

It takes a while to enable logging

Enabling logging and changing the settings takes about an hour. The first log object is saved to the bucket in a couple of hours after logging is enabled, but sometimes it may take longer to deliver logs.

When you change the target bucket, some logs are still delivered to the previous target bucket, and other logs are delivered to the new target bucket.

All changes to the settings will take effect without additional user actions.

Pricing

The standard Object Storage pricing rules apply to logging.

In this article: