Working with logs
You can monitor Data Proc cluster events in the management console. To do this, during creating or updating a cluster specify the log group in its settings.
If no log group is specified then the default log group for the folder of the cluster will be used. The default log group will be automatically created if there is no default log group in the folder.
Cluster logs are collected and displayed by Cloud Logging.
See details in section Logs in Data Proc.
Viewing cluster log entries
- Go to the folder page and select Data Proc.
- Click on the name of the cluster and select the Logs tab.
- (Optional) Specify the output settings:
-
-
Get job runner output Data Proc:
job_id="<dataproc_job_id>" -
Get YARN application containers stdout:
application_id="<YARN application_id>" AND yarn_log_type="stdout" -
Get stderr of a specific YARN container:
container_id="<YARN container_id>" AND yarn_log_type="stderr" -
Ger YARN Resource Manager logs:
hostname="<master host FQDN>" AND log_type="hadoop-yarn-resourcemanager"
-
-
Message logging levels: from
TRACEtoFATAL. -
Number of messages per page.
-
Message interval (pre-defined or arbitrary one).
-
If you don't have the Yandex.Cloud command line interface yet, install and initialize it.
Take a look at CLI command for collecting logs:
yc logging read --help
Examples:
-
Get logs of HDFS NameNode of the Data Proc cluster:
yc logging read \ --group-id "<log_group_id>" \ --resource-ids "<dataproc_cluster_id>" \ --filter "log_type=hadoop-hdfs-namenode" -
Get log messages in the last two hours for all Data Proc clusters, that sent logs to specified log group:
yc logging read \ --group-id "<log_group_id>" \ --resource-types "dataproc.cluster" \ --since 2h
Disabling log sending to the cloud
To disable sending logs to Cloud Logging, set Data Proc cluster property dataproc:disable_cloud_logging to true when creating or updating a cluster.
Storing logs
Receiving and storing logs is subject to pricing of the Cloud Logging. To change logs retention time and access control rules, edit the log group' settings.
Learn more about working with logs in the Yandex Cloud Logging documentation.