Logging actions with a bucket

Written by

Prerequisites
Format of the key for the log object
- Prefix of the key
Format of the log object
Logging specifics
- Best-effort log delivery
- It takes a while to enable logging
Pricing

In Object Storage, there is an option to log all actions with a bucket. You can record logs, for example, to run an internal security audit or get more granular information about bucket operations.

Logging is disabled by default. After you enable this option, Object Storage will write actions with the bucket to an object once an hour.

To save logs, do the following:

Define the source bucket that you want to log actions with.
Create a target bucket where you want to save the logs.
Enable logging.
(optional) Select the prefix of the object key.

Prerequisites

The source and target buckets must be in the same cloud.

Format of the key for the log object

Object Storage uses the following format of the key for the log object:

<prefix>/YYYY-MM-DD-HH-MM-SS-<ID>

Where:

prefix/: The prefix of the key for the log object. You can specify your own prefix when enabling logging.
YYYY-MM-DD-HH-MM-SS: Date and time of saving the log object in the target bucket (UTC format).
ID: A unique record ID that prevents the object from being overwritten.

Prefix of the key

The key prefix lets you distinguish:

Data belonging to different buckets, if the logs for multiple source buckets are saved to the same target bucket.
Logging actions from other actions with the bucket, if the logs are saved to the source bucket. That's because the logging operation is also considered an action with the bucket in this case.
The log objects from other objects, in order to delete logs from time to time. You can set up the lifecycle for the target bucket so that objects with a specific key prefix are deleted automatically.

Format of the log object

Logs are saved to a JSON file. For every action with the bucket, a record is written to the file in the following format:

Field	Type	Description
`bucket`	String	Bucket name.
`bytes_received`	Int64	Size of the request in bytes.
`bytes_send`	Int64	Response size in bytes.
`handler`	String	Request method in the `REST format.<HTTP method>.<subject>`.
`http_referer`	String	URL of the request source.
`ip`	String	User's IP address.
`method`	String	HTTP request method.
`object_key`	String	The object's key in URL encoded format.
`protocol`	String	Data transfer protocol version.
`range`	String	An HTTP header that defines the range of bytes to load from the object.
`requester`	String	User ID.
`request_args`	String	Arguments of the URL request.
`request_id`	String	Request ID.
`request_path`	String	Full path of the request.
`request_time`	Int64	Request processing time, in milliseconds.
`scheme`	String	Type of data transfer protocol. Acceptable values: - `http`, an application layer protocol. - `https`, an application layer protocol with encryption support.
`ssl_protocol`	String	Security protocol.
`status`	Int64	HTTP response code.
`storage_class`	String	Storage class of the object.
`timestamp`	String	Date and time of the operation with the bucket, in the `YYYY-MM-DDTHH:MM:MMZ` format.
`user_agent`	String	Client application (User Agent) that executed the request.
`version_id`	String	Version of the object.
`vhost`	String	Virtual host of the request. Acceptable values: – `storage.yandexcloud.net`. – `bucket name>.storage.yandexcloud.net`. – `website.yandexcloud.net`. – `<bucket name>.website.yandexcloud.net`.

Example of a record in the log file:

{
  "bucket": "my-bucket-example",
  "bytes_received": 749,
  "bytes_send": 1251,
  "handler": "REST.GET.OBJECT",
  "http_referer": "https://example.com/page",
  "ip": "84.201.121.46",
  "method": "GET",
  "object_key": "path/logs/2020-11-10-14-42-11-123f57b5-1853-4120-8d7a-5bcc1e9e9b4f",
  "protocol": "HTTP/1.1",
  "range": "-",
  "requester": "-",
  "request_args": "X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Date=20201030T072100Z\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Expires=43200\u0026X-Amz-Credential=ZGB4EY1...\u0026X-Amz-Signature=12f350...",
  "request_id": "1235efdab7445028",
  "request_path": "/my-bucket-example/path/logs/2020-11-10-14-42-11-123f57b5-1853-4120-8d7a-5bcc1e9e9b4f?X-Amz-...",
  "request_time":88,
  "scheme": "https",
  "ssl_protocol": "TLSv1.2",
  "status": 200,
  "storage_class": "STANDARD",
  "timestamp": "2020-11-10T13:21:18Z",
  "user_agent": "docker/19.03.9 go/go1.13.10 git-commit/1d238398e7 kernel/4.4.0-142-generic os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)",
  "version_id": "",
  "vhost": "storage.yandexcloud.net"
}

Logging specifics

There are several points to note about how actions with a bucket are logged in Object Storage.

Best-effort log delivery

Most requests to a bucket are written to the log file (if the bucket was set up correctly to support logging). Most records are written within a few hours after the request is actually processed.

However, Object Storage doesn't guarantee that the logs are saved in a complete and timely manner. It may take several hours to record an action with the bucket in a log file. In some cases, a record might fail to appear in the file.

The log file provides an overview of the nature of traffic in the bucket, but is not intended for logging every request. In the payment documents, you can find several requests that are not saved in the log file.

It takes a while to enable logging

Enabling logging and changing the settings takes about an hour. The first log object is saved to the bucket in a couple of hours after logging is enabled, but sometimes it may take longer to deliver logs.

When you change the target bucket, some logs are still delivered to the previous target bucket, and other logs are delivered to the new target bucket.

All changes to the settings will take effect without additional user actions.

Pricing

The standard Object Storage pricing rules apply to logging.