[/
    Copyright (c) 2019-2024 Ruben Perez Hidalgo (rubenperez038 at gmail dot com)
   
    Distributed under the Boost Software License, Version 1.0. (See accompanying
    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
]

[section:sql_formatting (Experimental) Client-side SQL query formatting]
[nochunk]

When issuing queries that contain untrusted input, prepared statement are usually
the way to go. However, some use cases like dynamic filters or batch inserts
are not attainable using them.

The functions described in this section can be used to compose SQL query
strings dynamically client-side while keeping your application secure.

[note
    This feature is experimental. Its API may change in subsequent releases.
]



[heading Common use cases]

* Queries involving several [*dynamic filters].
  See [link mysql.examples.dynamic_filters this example].
* [*Batch inserts]. Inserting rows one by one can lead to poor efficiency.
  You can use client-side query formatting to compose a single `INSERT` that
  performs several insertions at once. See [link mysql.examples.batch_inserts this example]
  for simple batch inserts and [link mysql.examples.batch_inserts_generic this other] for
  a generic utility to implement this case.
* [*PATCH-like updates]. If your query must update fields dynamically, based on some input,
  prepared statements may not be suitable. See [link mysql.examples.patch_updates this example].
* Queries involving [*dynamic identifiers], like table and field names.
* Performing [*conditional sorting].
* Any other use case that involves SQL that can change dynamically.

This feature can also be used to improve efficiency, as text queries perform less
round-trips to the server. See [link mysql.sql_formatting.efficiency this section]
for more info.


[heading Security considerations]

Client-side SQL formatting [*does protect against SQL injection] by appropriately quoting and escaping
string values. However, this feature [*does not understand your queries]. You need to make sure that
your formatting operations result in valid SQL. If you try to build very complex
queries and make mistakes, attackers may exploit logic errors in your formatting logic to overtake your system.



[heading Formatting simple queries]

You can use [reflink format_sql] to generate a SQL query from a format string
and a set of parameters, using a notation similar to `std::format`:

[sql_formatting_simple]

[reflink format_sql] doesn't involve communication with the server.
In order to work, it requires a [reflink format_options] instance describing
connection configuration, like the character set currently in use.
[refmem any_connection format_opts] provides an easy way to retrieve these.

[note
  Getting an `unknown_character_set` error? Have a look at [link mysql.sql_formatting.unknown_character_set this section].
]

All fundamental types can be used with query formatting. This includes integers, floating point types,
strings, blobs, dates and times:

[sql_formatting_other_scalars]

`std::optional<T>` and `boost::optional<T>` can also be used:

[sql_formatting_optionals]

Collections and ranges are supported, as long as its elements can be formatted:

[sql_formatting_ranges]

See [link mysql.sql_formatting.ranges this section] for more on formatting ranges,
and [link mysql.sql_formatting.reference this table] for a reference of types
that have built-in support for SQL formatting.

[note
  Like with `std::format`, the format string passed to `format_sql` must be known at
  compile-time. You can skip this check using the [reflink runtime] function.
]

Like `std::format`, you can use arguments with explicit indices:

[sql_formatting_manual_indices]

You can also use named arguments, using the initializer list overload:

[sql_formatting_named_args]

See [link mysql.sql_formatting_advanced.format_string_syntax this section] for an in-depth
explanation on format strings.





[heading Format specifiers]

Some types, like strings, can be formatted in multiple ways. As with
`std::format`, you can select how to format them using format specifiers.

As we've seen, strings are formatted as single-quoted values by default.
If you use the `{:i}` specifier, you can obtain dynamic SQL identifiers, instead:

[sql_formatting_specifiers]

Specifiers are compatible with explicit indices and named arguments, too.
This is equivalent to the previous snippet:

[sql_formatting_specifiers_explicit_indices]



[heading Building SQL strings incrementally using format_sql_to]

Some use cases, usually involving conditionals, may not be
expressible in terms of a single format string. In such cases, you can
use [reflink format_context] and [reflink format_sql_to] to
build query strings incrementally:

[sql_formatting_incremental_fn]
[sql_formatting_incremental_use]






[heading:ranges Formatting ranges with sequence]

The [reflink sequence] function can be used when the default range formatting isn't sufficient.
If the elements in your range are not formattable, you can pass a user-defined function to `sequence`
describing how to format each element:

[sql_formatting_sequence_1]

By default, elements are separated by commas, but this is configurable:

[sql_formatting_sequence_2]




[heading:unknown_character_set Solving the unknown_character_set error]

If you are getting a `boost::system::system_error` with a
`client_errc::unknown_character_set` error code (or getting this error code by other means),
your connection is currently unaware of the character set it's using, which is required by format operations.
Try the following:

* Make sure that you are performing connection establishment ([refmem any_connection connect] or
  [refmemunq any_connection async_connect]) before calling [refmemunq any_connection format_opts].
* Use [refmem any_connection set_character_set] or [refmemunq any_connection async_set_character_set]
  to set your connection's character set instead of using raw SQL.
* Some [refmem connect_params connection_collation] values are not supported by all servers
  and often trigger fallback behavior. If you are using a non-UTF8 character set, prefer setting it
  explicitly using [refmemunq any_connection set_character_set] or [refmemunq any_connection async_set_character_set].
  Don't rely on [refmem connect_params connection_collation].
* [refmem any_connection reset_connection] and [refmemunq any_connection async_reset_connection] wipe
  character set information. Call [refmemunq any_connection set_character_set] or [refmemunq any_connection async_set_character_set]
  after resetting your connection.

[warning
  [*Security considerations]: don't craft [reflink format_options] values manually.
  Always use [refmem any_connection format_opts].
]

For an explanation on why [reflink format_options] is necessary and how
character set tracking works, please read [link mysql.sql_formatting_advanced.format_options this section].




[heading Solving the invalid_encoding error]

SQL formatting can fail if you provide values that can't be securely formatted.
The most common cause is passing string values that are not valid according to the
passed character set. This triggers a `client_errc::invalid_encoding` error:

[sql_formatting_invalid_encoding]

You can validate your strings beforehand or handle the error once
it happened and reject the input. Other types may also produce format errors.

[tip
    If you prefer handling errors with error codes, instead of exceptions,
    use [reflink format_sql_to]. Please read
    [link mysql.sql_formatting_advanced.error_handling this section] for details.
]





[heading:efficiency Efficiency considerations]

Both client-side SQL formatting and prepared statements have pros and cons efficiency-wise:

* Client-formatted SQL entails [*less round-trips to the server]. For prepared statements, you usually need
  a call to prepare the statement, another one to execute it, and possibly a final one to close it.
  Client-formatted SQL only requires the execution round-trip. This performance gain increases with network
  latency and if you are using TLS.
* Prepared statements always entail a [*mutation of session state], while client-formatted SQL may not.
  If you're using a [reflink connection_pool] with prepared statements, you can't use
  [refmem pooled_connection return_without_reset], as this will leak the statement.
  With client-formatted queries, reset may not be required if your SQL doesn't mutate session state.
* Client-formatted SQL queries use a usually [*less efficient text-based protocol], while prepared statements
  use a more compact binary protocol. This is relevant if you're retrieving lots of data that is
  slow to convert to and from text (like doubles).
* [*Prepared statements can be re-used]. If you need to execute a query several times,
  prepared statements will only be parsed once.
* Client-formatted SQL allows [*more efficient patterns] than prepared statements,
  like batch inserts.
* You can use client-formatted SQL to generate several [*semicolon-separated queries]
  to be run in batch.



[heading:reference Types with built-in support for SQL formatting]

[table
    [
        [C++ type]
        [Formatted as...]
        [Example]
    ]
    [
        [`signed char`, `short`, `int`, `long`, `long long`]
        [
            Integral literal[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_signed]
        ]
    ]
    [
        [`unsigned char`, `unsigned short`, `unsigned int`, `unsigned long`, `unsigned long long`]
        [
            Integral literal[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_unsigned]
        ]
    ]
    [
        [`bool`]
        [
            Integral literal `1` if `true`, `0` if `false`[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_bool]
        ]
    ]
    [
        [
            String types (convertible to [reflink string_view]), including:[br][br]
            `std::string`[br][br]
            [reflink string_view][br][br]
            `std::string_view`[br][br]
            `const char*`[br][br]
        ]
        [
            Without format specifiers: single-quoted escaped string literal. Note that `LIKE` special characters (`%` and `_`) are not escaped.[br][br]
            [*`i`] format specifier: backtick-quoted, escaped SQL identifier.[br][br]
            [*`r`] format specifier: raw, unescaped SQL. [*Warning]: use this specifier with caution.
        ]
        [
            [sql_formatting_reference_string]
        ]
    ]
    [
        [
            Blob types (convertible to `span<const unsigned char>`), including:[br][br]
            [reflink blob] (`std::vector<unsigned char>`)[br][br]
            [reflink blob_view] (`span<const unsigned char>`)[br][br]
            `std::array<unsigned char, N>`
        ]
        [
            Hex string literal[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_blob]
        ]
    ]
    [
        [`float`, except NaN and inf]
        [
            Floating-point literal, after casting to `double.`[br]
            MySQL does not support NaNs and infinities. Attempting to format
            these cause a `client_errc::unformattable_value` error.[br]
            No format specifiers allowed.
        ]
        [
            [sql_formatting_reference_float]
        ]
    ]
    [
        [`double`, except NaN and inf]
        [
            Floating-point literal.[br]
            MySQL does not support NaNs and infinities. Attempting to format
            these cause a `client_errc::unformattable_value` error.[br]
            No format specifiers allowed.
        ]
        [
            [sql_formatting_reference_double]
        ]
    ]
    [
        [[reflink date]]
        [
            Single quoted, `DATE`-compatible string literal[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_date]
        ]
    ]
    [
        [[reflink datetime]]
        [
            Single quoted `DATETIME`-compatible string literal[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_datetime]
        ]
    ]
    [
        [[reflink time] and `std::chrono::duration` types convertible to [reflink time]]
        [
            Single quoted `TIME`-compatible string literal[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_time]
        ]
    ]
    [
        [`std::nullptr_t`]
        [
            `NULL`[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_nullptr]
        ]
    ]
    [
        [
            `boost::optional<T>` and `std::optional<T>`, `T` being one of the fundamental types above.[br]
            Not applicable to custom types or ranges.[br]
            No format specifiers allowed
        ]
        [
            Formats the underlying value if there is any.[br]
            `NULL` otherwise.[br]
        ]
        [
            [sql_formatting_reference_optional]
        ]
    ]
    [
        [[reflink field] and [reflink field_view]]
        [
            Formats the underlying value.[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_field]
        ]
    ]
    [
        [
            Range of formattable elements. Informally, such ranges support
            `std::begin()` and `std::end()`, and its iterator `operator*`
            must yield one of the following:

                * A [reflink2 WritableFieldTuple WritableField] (i.e. one of the fundamental types above).
                * A type with a custom formatter.

            Ranges of ranges are not supported.
            Note that `vector<unsigned char>` and similar types are formatted as blobs, not as sequences.
            
            See [reflink2 Formattable the Formattable concept reference] for a formal definition.
        ]
        [
            Formats each element in the range, separating elements with commas.[br]
            Specifiers can be applied to individual elements by prefixing them with a colon (`:`)
        ]
        [
            [sql_formatting_reference_ranges]
        ]
    ]
    [
        [
            [reflink format_sequence_view] (as returned by [reflink sequence])
        ]
        [
            Formats each element in a range by calling a user-supplied function,
            separating elements by a glue string (a comma by default).[br]
            No format specifiers allowed
        ]
        [
            [sql_formatting_reference_sequence]
        ]
    ]
    [
        [Custom type that specializes [reflink formatter]]
        [
            Calls `formatter::parse` and `formatter::format`[br]
            May accept user-defined format specifiers.
        ]
        []
    ]
    [
        [[reflink formattable_ref]]
        [
            Formats the underlying value. Can represent any of the types above.[br]
            Accepts the same format specifiers as the underlying type.
        ]
        [
            [sql_formatting_reference_formattable_ref]
        ]
    ]
]

[endsect]