Deleting all objects in a bucket

Written by

Yandex Cloud

To clear a bucket and not pay for storage:

AWS CLI

Python 3 (boto3)

Create a variable containing the bucket name:
```
BUCKET_NAME=<bucket name>
```

Delete all versions of objects from the bucket:

aws s3api delete-objects \
  --endpoint-url https://storage.yandexcloud.net \ 
  --bucket $BUCKET_NAME \
  --delete \
    "$(aws s3api list-object-versions \
      --endpoint-url https://storage.yandexcloud.net \
      --bucket $BUCKET_NAME \
      --query '{Objects: Versions[].{Key: Key, VersionId: VersionId}}' \
      --max-items 1000)"

Command output:

{
    "Deleted": [
        {
            "Key": "object_000",
            "VersionId": "0005CDD3521954A6"
        },
        {
            "Key": "object_001",
            "VersionId": "0005CDD3523B6ADB"
        },
        ...
    ]
}

With this command, you can delete up to 1000 versions of objects: this is related to restrictions for the aws s3api delete-objects operation. If there are more versions in the bucket, run the command several times.

If the bucket has versioning enabled, delete all the delete markers:

aws s3api delete-objects \
  --endpoint-url https://storage.yandexcloud.net \
  --bucket $BUCKET_NAME \
  --delete \
    "$(aws s3api list-object-versions \
      --endpoint-url https://storage.yandexcloud.net \
      --bucket $BUCKET_NAME \
      --query '{Objects: DeleteMarkers[].{Key: Key, VersionId: VersionId}}' \
      --max-items 1000)"

Command output:

{
    "Deleted": [
        {
            "Key": "object_034",
            "DeleteMarker": true,
            "DeleteMarkerVersionId": "0005CDD35274A467"
        },
        {
            "Key": "object_057",
            "DeleteMarker": true,
            "DeleteMarkerVersionId": "0005CDD3528B1BC3"
        },
        ...
    ]
}

With this command, you can delete up to 1000 delete markers: this is related to restrictions for the aws s3api delete-objects operation. If there are more versions in the bucket, run the command several times.

Delete partially uploaded objects:

aws s3api list-multipart-uploads \
  --endpoint-url https://storage.yandexcloud.net \ 
  --bucket $BUCKET_NAME \
| jq -r '.Uploads[] | "--key \"\(.Key)\" --upload-id \(.UploadId)"' \
| while read -r line; do
  eval 
    "aws s3api abort-multipart-upload \
      --endpoint-url https://storage.yandexcloud.net \
      --bucket $BUCKET_NAME \
      $line";
done

Get a list of object parts remaining in the bucket:

aws s3api list-multipart-uploads \
  --endpoint-url https://storage.yandexcloud.net \ 
  --bucket $BUCKET_NAME \
| jq -r '.Uploads[] | "--key \"\(.Key)\" --upload-id \(.UploadId)"' \
| while read -r line; do
  eval 
    "aws s3api list-parts \
      --endpoint-url https://storage.yandexcloud.net \
      --bucket $BUCKET_NAME \
      $line";
done

The list may contain parts of objects that began uploading before and finished uploading after the previous step executed. If the list is not empty, repeat steps 4 and 5.

Run the following code:

import boto3

bucket_name = '<bucket name>'
s3 = boto3.resource('s3',
    endpoint_url='https://storage.yandexcloud.net',
    aws_access_key_id='<key ID>',
    aws_secret_access_key='<secret key>')
bucket = s3.Bucket(bucket_name)

# Deleting all versions (works for non-versioned buckets too).
bucket.object_versions.delete()

# Aborting all multipart uploads, which also deletes all parts.
for multipart_upload in bucket.multipart_uploads.iterator():
    # Part uploads that are currently in progress may or may not succeed,
    # so it might be necessary to abort a multipart upload multiple times.
    while len(list(multipart_upload.parts.all())) > 0:
        multipart_upload.abort()

Deleting all objects in a bucket

Was the article helpful?