Logging actions with the bucket
To track operations with the bucket, enable logging. Detailed information about requests to the source bucket will be saved in an object in the target bucket. However, Object Storage does not guarantee that the logs are saved in a complete and timely manner.
By default, logging is disabled. After you enable it, Object Storage will save information about actions with the bucket once an hour.
Enable logging
To log requests to the bucket:
-
Use an existing target bucket or create a new one. This is the bucket your logs will be written to.
How to create a bucketThe minimum role required to create a bucket is
storage.editor
. See the role description for details.Management consoleYandex Cloud CLIAWS CLITerraformAPI- In the management console
, select the folder where you want to create a bucket. - Select Object Storage.
- Click Create bucket.
- On the bucket creation page:
-
Enter a name for the bucket according to the naming requirements.
By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.
-
Limit the maximum bucket size, if required.
If the value is
0
, the maximum size is not limited and is similar to the enabled No limit option. -
Select the type of access.
-
Select the default storage class.
-
Add labels, if needed:
- Click Add label.
- Enter a label in
key: value
format. - Click Enter.
-
Click Create bucket to complete the operation.
-
If you don't have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the
--folder-name
or--folder-id
parameter.-
View a description of the CLI command to create a bucket:
yc storage bucket create --help
-
Create a bucket in the default folder:
yc storage bucket create \ --name <bucket_name> \ --default-storage-class <storage_class> \ --max-size <maximum_bucket_size> \ --public-read \ --public-list \ --public-config-read
Where:
-
--name
: Bucket name. For more information, see Bucket naming rules.By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.
-
--default-storage-class
: Storage class. The possible values include:standard
: Standard storagecold
: Cold storageice
: Ice storage
-
--max_size
: Maximum bucket size (bytes)0
: No limit -
--public-read
: Flag to enable public read access to bucket objects. -
--public-list
: Flag to enable public access to view the list of bucket objects. -
--public-config-read
: Flag to enable public read access to the bucket configuration.
The
name
parameter is required. Other parameters are optional. By default, the--max-size
parameter is set to0
and public access to the bucket is disabled.Result:
name: <bucket_name> folder_id: b1gmit33ngp6******** anonymous_access_flags: read: false list: false default_storage_class: STANDARD versioning: VERSIONING_DISABLED max_size: "53687091200" acl: {} created_at: "2022-12-16T14:05:12.196007Z"
-
If you do not have the AWS CLI yet, install and configure it.
In the terminal, run the command below by specifying the bucket name and the Object Storage endpoint:
aws --endpoint-url=https://storage.yandexcloud.net \ s3 mb s3://<bucket_name>
Note
By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage. For more information, see Bucket naming rules.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
-
In the configuration file, describe the parameters of the resources you want to create:
terraform { required_providers { yandex = { source = "yandex-cloud/yandex" } } required_version = ">= 0.13" } provider "yandex" { token = "<IAM_or_OAuth_token>" cloud_id = "<cloud_ID>" folder_id = "<folder_ID>" zone = "ru-central1-a" } resource "yandex_iam_service_account" "sa" { name = "<service_account_name>" } // Assigning roles to the service account resource "yandex_resourcemanager_folder_iam_member" "sa-editor" { folder_id = "<folder_ID>" role = "storage.editor" member = "serviceAccount:${yandex_iam_service_account.sa.id}" } // Creating a static access key resource "yandex_iam_service_account_static_access_key" "sa-static-key" { service_account_id = yandex_iam_service_account.sa.id description = "static access key for object storage" } // Creating a bucket using the key resource "yandex_storage_bucket" "test" { access_key = yandex_iam_service_account_static_access_key.sa-static-key.access_key secret_key = yandex_iam_service_account_static_access_key.sa-static-key.secret_key bucket = "<bucket_name>" }
Where:
yandex_iam_service_account
: Description of the service account that will create and use a bucket:name
: Service account name
yandex_storage_bucket
: Bucket description:bucket
: Bucket name
For more information about resources you can create using Terraform, see the provider documentation
. -
Make sure the configuration files are valid.
- In the command line, go to the directory where you created the configuration file.
- Run a check using this command:
terraform plan
If the configuration is described correctly, the terminal will display a list of created resources and their parameters. If the configuration contains any errors, Terraform will point them out.
-
Deploy cloud resources.
-
If the configuration does not contain any errors, run this command:
terraform apply
-
Confirm that you want to create the resources.
All the resources you need will then be created in the specified folder. You can check the new resources and their configuration using the management console
. -
To create a bucket, use the create REST API method for the Bucket resource, the BucketService/Create gRPC API call, or the create S3 API method.
- In the management console
Note
To write logs to the target bucket with an access policy configured, add there a rule to allow any account to perform the PutObject
action, and specify the log object key prefix, e.g., logs/
, as a resource.
-
Enable logging in the source bucket that you want to track.
AWS CLITerraformAPITo enable logging via the AWS CLI:
-
Create a file with logging settings in JSON format, e.g.:
{ "LoggingEnabled": { "TargetBucket": "<target_bucket_name>", "TargetPrefix": "<key_prefix>" } }
Where:
TargetBucket
: Name of the target bucket to write logs to.TargetPrefix
: Prefix of the key used for log objects, e.g.,logs/
.
-
Enable logging in the bucket:
aws s3api put-bucket-logging \ --bucket <source_bucket_name> \ --endpoint-url https://storage.yandexcloud.net \ --bucket-logging-status file://<path_to_settings_file>
Where:
--bucket
: Name of the source bucket to enable action logging for.--bucket-logging-status
: Path to the logging settings file.
With Terraform
, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. They store the infrastructure description in HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License .For more information about the provider resources, see the documentation on the Terraform
website or mirror website .If you change the configuration files, Terraform automatically determines which part of your configuration is already deployed and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
To enable logging for a bucket that you wish to monitor:
-
Open the Terraform configuration file and add a section called
logging
to the fragment describing the bucket.resource "yandex_storage_bucket" "log_bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<name_of_bucket_to_store_logs>" } resource "yandex_storage_bucket" "bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<source_bucket_name>" acl = "private" logging { target_bucket = yandex_storage_bucket.log_bucket.id target_prefix = "log/" } }
Where:
access_key
: ID of the static access key.secret_key
: Value of the secret access key.target_bucket
: Reference to the bucket that will store logs.target_prefix
: Prefix of the key used for log objects, e.g.,logs/
.
For more information about
yandex_storage_bucket
resource parameters in Terraform, see the provider documentation .
-
In the terminal, change to the folder where you edited the configuration file.
-
Make sure the configuration file is correct using the command:
terraform validate
If the configuration is correct, the following message is returned:
Success! The configuration is valid.
-
Run the command:
terraform plan
The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.
-
Apply the configuration changes:
terraform apply
-
Confirm the changes: type
yes
in the terminal and press Enter.
All the resources you need will then be created in the specified folder. You can check the new resources and their configuration using the management console
.To enable logging for your bucket, use the putBucketLogging S3 API method.
Example of the HTTP request body:
<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01"> <LoggingEnabled> <TargetBucket>target bucket name</TargetBucket> <TargetPrefix>logs/</TargetPrefix> </LoggingEnabled> </BucketLoggingStatus>
Where:
<TargetBucket>
: Target bucket name.<TargetPrefix>
: Prefix of the key used for log objects, e.g.,logs/
. This is an optional parameter.
-
Get the logging settings
To get the name of the target bucket and the prefix of the key for the log object, follow these steps:
To retrieve the logging settings via the AWS CLI:
-
Run this command:
aws s3api get-bucket-logging \ --bucket <bucket_name> \ --output json \ --endpoint-url https://storage.yandexcloud.net
Result:
{ "LoggingEnabled": { "TargetBucket": "<bucket_name>", "TargetPrefix": "<key_prefix>" } }
Use the getBucketLogging S3 API method.
Example of the HTTP response body:
HTTP/1.1 200
<?xml version="1.0" encoding="UTF-8"?>
<BucketLoggingStatus>
<LoggingEnabled>
<TargetBucket>target bucket name</TargetBucket>
<TargetPrefix>logs/</TargetPrefix>
</LoggingEnabled>
</BucketLoggingStatus>
Get the logs
To get logs, download the object prefixed with logs/
from the target bucket:
- In the management console
, select the folder where the target bucket with logs is located. - Select Object Storage.
- Click the name of the bucket with the logs.
- Click on the object name with the
logs/
prefix. - Click Download.
Disable logging
To disable logging, follow these steps:
With Terraform
For more information about the provider resources, see the documentation on the Terraform
If you change the configuration files, Terraform automatically determines which part of your configuration is already deployed and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
To disable the logging mechanism:
-
Open the Terraform configuration file and delete the
logging
section from the fragment describing the bucket.Example bucket description in a Terraform configuration... resource "yandex_storage_bucket" "log_bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<name_of_bucket_to_store_logs>" } resource "yandex_storage_bucket" "bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<source_bucket_name>" acl = "private" logging { target_bucket = yandex_storage_bucket.log_bucket.id target_prefix = "log/" } } ...
-
In the terminal, change to the folder where you edited the configuration file.
-
Make sure the configuration file is correct using the command:
terraform validate
If the configuration is correct, the following message is returned:
Success! The configuration is valid.
-
Run the command:
terraform plan
The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.
-
Apply the configuration changes:
terraform apply
-
Confirm the changes: type
yes
in the terminal and press Enter.
You can verify the changes in the management console
Use the putBucketLogging S3 API method. In the request body, send the <BucketLoggingStatus>
parameter with an empty value.
Example of the HTTP request body:
<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01" />