Yandex Cloud
  • Services
  • Solutions
  • Why Yandex Cloud
  • Pricing
  • Documentation
  • Contact us
Get started
Language / Region
© 2022 Yandex.Cloud LLC
  • Contents
  • Getting started
    • Overview
    • Creating a database
    • Authentication
    • YDB command line interface (CLI)
    • YQL query language
    • YDB SDK
    • Self-deployment
      • Overview
      • Docker
      • Binary file
      • Minikube
    • Useful links
    • Amazon DynamoDB-compatible Document API
      • Setting up AWS tools
      • Working with data from the HTTP interface
      • Working with the AWS CLI
        • Overview
        • Creating a table
        • Adding data to a table
        • Reading data from a table
        • Updating data
        • Data selections
        • Deleting created resources
      • Working with the AWS SDK
        • Overview
        • Creating a table
        • Uploading data to a table
        • Managing records in a table
          • Creating a record
          • Reading a record
          • Updating a record
          • Deleting a record
        • Searching and extracting data
        • Deleting a table
  • Practical guidelines
    • Deploying a web application
    • Developing a Slack bot
    • Connecting to YDB from a Python function in Yandex Cloud Functions
    • Connecting to a YDB database from a Yandex Cloud Functions function in Node.js
    • Converting a video to a GIF in Python
    • Developing a skill for Alice and a website with authorization
  • Concepts
    • Overview
    • Terms and definitions
    • Connecting to and authenticating with a database
    • Data model and schema
    • Serverless and Dedicated operation modes
    • Transactions
    • Secondary indexes
    • Time to Live (TTL)
    • Scan queries
    • Database limits
    • YDB cluster
      • Overview
      • General YDB schema
      • Disk subsystem of a cluster
    • Quotas and limits
  • Step-by-step instructions
    • Overview
    • Renaming
    • Using a query plan and AST
    • Reading and writing data
    • Working with secondary indexes
  • Recommendations
    • Overview
    • Schema design
    • Partitioning tables
    • Secondary indexes
    • Paginated output
    • Loading large data volumes
    • Using timeouts
  • Managing databases
    • Overview
    • Cloud management console
      • Overview
      • Creating, updating, and deleting databases
      • Tables and directories
      • Access management
    • Yandex.Cloud CLI
    • Backup and recovery
    • Diagnostics
      • Overview
      • System views
      • Monitoring
    • Metric reference
  • Pricing policy
    • Overview
    • Serverless mode
      • Pricing policy for serverless mode
      • Query cost for YQL
      • Request cost for the Document API
      • Request cost for special APIs
    • Dedicated mode
  • Amazon DynamoDB-compatible HTTP API
    • All methods
    • Actions
      • BatchGetItem
      • BatchWriteItem
      • CreateTable
      • DeleteItem
      • DeleteTable
      • DescribeTable
      • DescribeTimeToLive
      • GetItem
      • ListTables
      • PutItem
      • Query
      • Scan
      • TransactGetItems
      • TransactWriteItems
      • UpdateItem
      • UpdateTimeToLive
    • Common errors
  • YQL
    • Overview
    • Data types
      • Overview
      • Simple
      • Optional
      • Containers
      • Special
      • Type casting
      • Text representation of data types
      • JSON
    • Syntax
      • Overview
      • Lexical structure
      • Expressions
      • ACTION
      • ALTER TABLE
      • CREATE TABLE
      • DECLARE
      • DELETE
      • DISCARD
      • DROP TABLE
      • GROUP BY
      • FLATTEN
      • INSERT
      • INTO RESULT
      • JOIN
      • PRAGMA
      • REPLACE
      • SELECT
      • UPDATE
      • UPSERT
      • VALUES
      • WINDOW
      • Unsupported statements
    • Built-in functions
      • Overview
      • Basic
      • Aggregate
      • Window
      • For lists
      • For dictionaries
      • For structures
      • For types
      • For JSON
      • C++ libraries
        • Overview
        • Hyperscan
        • Pcre
        • Pire
        • Re2
        • String
        • Unicode
        • DateTime
        • Url
        • Ip
        • Yson
        • Digest
        • Math
        • Histogram
    • YQL tutorial
      • Overview
      • Creating a table
      • Adding data to a table
      • Selecting data from all columns
      • Selecting data from specific columns
      • Sorting and filtering
      • Data aggregation
      • Additional selection criteria
      • Joining tables with JOIN
      • Inserting and updating data with REPLACE
      • Inserting and updating data with UPSERT
      • Inserting data with INSERT
      • Updating data with UPDATE
      • Deleting data
      • Adding and deleting columns
      • Deleting a table
  • Working with the YDB CLI
    • Overview
    • Install
    • Structure of YDB CLI commands
    • Service commands
    • Connecting to and authenticating with a database
    • Global parameters
    • Working with the DB schema
      • List of objects
      • Information about the object
      • Directories
      • Secondary indexes
      • Renaming tables
    • Operations with data
      • Making a DB query
      • Query execution plan
      • Streaming table reads
      • Scan queries
    • Importing and exporting data
      • Overview
      • File structure of data export
      • Exporting data to the file system
      • Importing data from the file system
      • Connecting to and authenticating with S3
      • Exporting data to S3
      • Importing data from S3
    • Managing profiles
      • Overview
      • Creating a profile
      • Using a profile in requests
      • Getting profile information
      • Deleting a profile
      • Activated profile
    • Information services
      • List of endpoints
      • Authentication
    • Load testing
      • Overview
      • Stock load
  • Working with the YDB SDK
    • Overview
    • Install
    • Authentication
    • Test app
      • Overview
      • C++
      • C# (.NET)
      • Go
      • Java
      • Node.js
      • PHP
      • Python
      • Archive
        • Go v1
        • Go v2
    • Handling errors in the API
    • Code recipes
      • Overview
      • Authentication
        • Overview
        • Using a token
        • Anonymous
        • Service account file
        • Metadata service
        • Using environment variables
        • Username and password based
      • Balancing
        • Overview
        • Random choice
        • Prefer the nearest data center
        • Prefer the availability zone
      • Running repeat queries
      • Troubleshooting
        • Overview
        • Enable logging
        • Enable metrics in Prometheus
        • Enable tracing in Jaeger
      • Setting the session pool size
  • Managing a cluster
    • Overview
    • Kubernetes
      • Overview
      • Deploying in Yandex Managed Service for Kubernetes
      • Deploying in AWS Elastic Kubernetes Service
      • Use
    • Manual
      • Overview
      • Local deployment
      • Cluster configuration
      • Production checklist
      • Maintaining a cluster's disk subsystem
        • Overview
        • How to stay within the failure model
        • Disk load balancing
        • Methods to free up space on physical devices
        • Cluster extension
        • Adding storage groups
        • Safe restart and shutdown of nodes
        • Enabling/disabling SelfHeal
        • Enabling/disabling Scrubbing
        • Moving VDisks
        • Updating configurations via CMS
        • Updating configuration of the actor system
    • Embedded UI
      • Overview
      • YDB Monitoring
      • Hive web-viewer
      • Connections overview
      • Logs
      • Charts
    • System views
  • Questions and answers
    • Overview
    • General questions
    • Errors
    • YQL
    • Serverless
    • All questions on one page
  1. YQL
  2. Built-in functions
  3. C++ libraries
  4. Url

Url

Written by
Yandex Cloud
  • Normalize
  • NormalizeWithDefaultHttpScheme
  • Encode / Decode
  • Parse
  • Get...
  • Cut...
  • ...Punycode...
  • ...Query...

Normalize

  • Url::Normalize(String) -> String?

Normalizes the URL in a robot-friendly way: converts the hostname into lowercase, strips out certain fragments, and so on.
The normalization result only depends on the URL itself. The normalization DOES NOT include operations depending on the external data: transformation based on duplicates, mirrors, etc.

Returned value:

  • Normalized URL.
  • NULL, if the passed string argument can't be parsed as a URL.

Examples

SELECT Url::Normalize("hTTp://wWw.yAnDeX.RU/"); -- "http://www.yandex.ru/"
SELECT Url::Normalize("http://ya.ru#foo");      -- "http://ya.ru/"

NormalizeWithDefaultHttpScheme

  • Url::NormalizeWithDefaultHttpScheme(String?) -> String?

Normalizes similarly to Url::Normalize, but inserts the http:// schema in case there is no schema.

Returned value:

  • Normalized URL.
  • Source URL, if the normalization has failed.

Examples

SELECT Url::NormalizeWithDefaultHttpScheme("wWw.yAnDeX.RU");    -- "http://www.yandex.ru/"
SELECT Url::NormalizeWithDefaultHttpScheme("http://ya.ru#foo"); -- "http://ya.ru/"

Encode / Decode

Encode a UTF-8 string to the urlencoded format (Url::Encode) and back (Url::Decode).

List of functions

  • Url::Encode(String?) -> String?
  • Url::Decode(String?) -> String?

Examples

SELECT Url::Decode("http://ya.ru/%D1%81%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D0%B0"); 
  -- "http://ya.ru/page"
SELECT Url::Encode("http://ya.ru/page");                                         
  -- "http://ya.ru/%D1%81%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D0%B0"

Parse

Parses the URL into parts.

  • Url::Parse(Parse{Flags:AutoMap}) -> Struct< Frag: String?, Host: String?, ParseError: String?, Pass: String?, Path: String?, Port: String?, Query: String?, Scheme: String?, User: String? >

Examples

SELECT Url::Parse(
  "https://en.wikipedia.org/wiki/Isambard_Kingdom_Brunel?s=24&g=h-24#Great_Western_Railway");
/*
(
  "Frag": "Great_Western_Railway",
  "Host": "en.wikipedia.org",
  "ParseError": null,
  "Pass": null,
  "Path": "/wiki/Isambard_Kingdom_Brunel",
  "Port": null,
  "Query": "s=24&g=h-24",
  "Scheme": "https",
  "User": null
)
*/

Get...

Get a component of the URL.

List of functions

  • Url::GetScheme(String{Flags:AutoMap}) -> String

  • Url::GetHost(String?) -> String?

  • Url::GetHostPort(String?) -> String?

  • Url::GetSchemeHost(String?) -> String?

  • Url::GetSchemeHostPort(String?) -> String?

  • Url::GetPort(String?) -> String?

  • Url::GetTail(String?) -> String? -- everything following the host: path + query + fragment

  • Url::GetPath(String?) -> String?

  • Url::GetFragment(String?) -> String?

  • Url::GetCGIParam(String?, String) -> String? -- The second parameter is the name of the intended CGI parameter.

  • Url::GetDomain(String?, Uint8) -> String? -- The second parameter is the required domain level.

  • Url::GetTLD(String{Flags:AutoMap}) -> String

  • Url::IsKnownTLD(String{Flags:AutoMap}) -> Bool -- Registered on http://www.iana.org/

  • Url::IsWellKnownTLD(String{Flags:AutoMap}) -> Bool -- Belongs to a small whitelist of com, net, org, ru, and so on.

  • Url::GetDomainLevel(String{Flags:AutoMap}) -> Uint64

  • Url::GetSignificantDomain(String{Flags:AutoMap}, [List<String>?]) -> String
    Returns a second-level domain in most cases and a third-level domain for the hostnames like: ***.XXX.YY, where XXX is com, net, org, co, gov, or edu. You can redefine this list using an optional second argument

  • Url::GetOwner(String{Flags:AutoMap}) -> String
    Returns the domain that's most likely owned by an individual or organization. Unlike Url::GetSignificantDomain, it uses a special whitelist. Besides the ***.co.uk domains, it can return a third-level domain used by free hosting sites and blogs (for example: something.livejournal.com)s

Examples

SELECT Url::GetScheme("https://ya.ru");           -- "https://"
SELECT Url::GetDomain("http://www.yandex.ru", 2); -- "yandex.ru"

Cut...

  • Url::CutScheme(String?) -> String?
    Returns the passed URL without the schema (http://, https://, etc.).

  • Url::CutWWW(String?) -> String?
    Returns the passed domain without the "www." prefix (if any).

  • Url::CutWWW2(String?) -> String?
    Returns the passed domain without the prefixes like " www.", " www2.", " wwww777." (if any).

  • Url::CutQueryStringA­ndFragment(String{Flags:AutoMap}) -> String
    Returns a copy of the passed URL, stripping out all the CGI parameters and fragments ("?foo=bar" and/or "#baz").

Examples

SELECT Url::CutScheme("http://www.yandex.ru"); -- "www.yandex.ru"
SELECT Url::CutWWW("www.yandex.ru");           -- "yandex.ru"

...Punycode...

Punycode transformations.

List of functions

  • Url::HostNameToPunycode(String{Flag:AutoMap}) -> String?
  • Url::ForceHostNameToPunycode(String{Flag:AutoMap}) -> String
  • Url::PunycodeToHostName(String{Flag:AutoMap}) -> String?
  • Url::ForcePunycodeToHostName(String{Flag:AutoMap}) -> String
  • Url::CanBePunycodeHostName(String{Flag:AutoMap}) -> Bool

Examples

SELECT Url::PunycodeToHostName("xn--d1acpjx3f.xn--p1ai"); -- "яндекс.рф"

...Query...

Query transformations.

List of functions

Url::QueryStringToList(String{Flag:AutoMap}, [
  KeepBlankValues:Bool?,  -- Empty values in percent-encoded queries are interpreted as empty strings, defaults to false.
  Strict:Bool?,           -- If false, parsing errors are ignored and incorrect fields are skipped, defaults to true.
  MaxFields:Uint32?,      -- The maximum number of fields. If exceeded, an exception is thrown. Defaults to Max<Uint32>.
  Separator:String?       -- A key-value pair separator, defaults to '&'.
]) -> List<Tuple<String, String>>
Url::QueryStringToDict(String{Flag:AutoMap}, [
  KeepBlankValues:Bool?,  -- Empty values in percent-encoded queries are interpreted as empty strings, defaults to false.
  Strict:Bool?,           -- If false, parsing errors are ignored and incorrect fields are skipped, defaults to true.
  MaxFields:Uint32?,      -- The maximum number of fields. If exceeded, an exception is thrown. Defaults to Max<Uint32>.
  Separator:String?       -- A key-value pair separator, defaults to '&'.
]) -> Dict<String, List<String>>
Url::BuildQueryString(Dict<String, List<String?>>{Flag:AutoMap}, [
  Separator:String?       -- A key-value pair separator, defaults to '&'.
]) -> String
Url::BuildQueryString(Dict<String, String?>{Flag:AutoMap}, [
  Separator:String?       -- A key-value pair separator, defaults to '&'.
]) -> String
Url::BuildQueryString(List<Tuple<String, String?>>{Flag:AutoMap}, [
  Separator:String?       -- A key-value pair separator, defaults to '&'.
]) -> String

Examples

SELECT Url::QueryStringToList("a=1&b=2&a=3");                       -- [("a", "1"), ("b", "2"), ("a", "3")]
SELECT Url::QueryStringToDict("a=1&b=2&a=3");                       -- {"b" : ["2"], "a" : ["1", "3"]}
SELECT Url::BuildQueryString([("a", "1"), ("a", "3"), ("b", "2")]); -- "a=1&a=3&b=2"
SELECT Url::BuildQueryString({"a" : "1", "b" : "2"});               -- "b=2&a=1"
SELECT Url::BuildQueryString({"a" : ["1", "3"], "b" : ["2", "4"]}); -- "b=2&b=4&a=1&a=3"

Was the article helpful?

Language / Region
© 2022 Yandex.Cloud LLC
In this article:
  • Normalize
  • NormalizeWithDefaultHttpScheme
  • Encode / Decode
  • Parse
  • Get...
  • Cut...
  • ...Punycode...
  • ...Query...