2. Lexical structure

2.1 Source text and files

CS-LEX-001. Ordinary DirectSharp source uses .ds. Include fragments use .inc and must not be compiled independently.

CS-LEX-005. Source uses UTF-8. A BOM is permitted. LF and CRLF are semantically equivalent; a final newline is optional. Identifier restrictions do not restrict Unicode in comments or textual data.

CS-LEX-006. Indentation, blank lines and trailing whitespace do not determine scope. Tabs and spaces are accepted. Comments may separate tokens wherever whitespace is legal, but cannot join fragments into one token: cus/* note */tomer is not the identifier customer.

2.2 Identifiers and casing

CS-LEX-002. Ordinary identifiers match [A-Za-z_][A-Za-z0-9_]*. Binding is case-insensitive and culture-independent. Declarations in the same scope that differ only by case are duplicates.

CS-LEX-007. Prefixing an identifier with @ permits a keyword spelling to be used as a name. The prefix is not part of symbol identity and does not permit spaces, hyphens or Unicode identifiers.

let @class = 10
let customer = LoadCustomer()
let Customer = LoadAnotherCustomer() // invalid: duplicate name

customer, Customer and @Customer denote the same identifier. @some-name is not an escaped single name. Imported members whose names conflict with keywords use the same escape. Exotic CLR names outside this spelling model are not grounds to widen ordinary identifiers.

CS-LEX-003. Canonical formatting uses lowercase language keywords, camelCase locals/parameters, and normal .NET/application casing for types and members. Different keyword casing remains valid source.

CS-LEX-008. An indivisible multiword language token uses lowercase snake_case; independently meaningful grammar words remain separate. Examples include for_each, order_by, try_get, old_value_of and add_handler, alongside is not nothing, new Customer with and end if. Spelling normalization does not activate retired features.

2.3 Comments and block endings

CS-LEX-009. // runs to the physical line end. /* ... */ comments may nest, and the outer comment ends only when nesting depth returns to zero. An unterminated comment is a compile-time error. /// supplies documentation comments.

/* Temporarily disabled:
    /* Existing explanation. */
    Process(customer)
*/

Apostrophe, REM and # are not alternate comment introducers.

CS-LEX-004. Canonical examples prefer explicit matching endings. end is the closing keyword; its qualifier identifies the construct being closed. Bare end remains permitted where the construct grammar allows it. Do not fuse closers into end_if or end_for_each.

2.4 Statement boundaries

CS-LEX-010. Newline is the normal statement separator. Semicolon is an optional separator/terminator, including multiple statements on one line. A lone semicolon is a valid empty statement and creates no value or scope. Blank/comment-only lines create no statements.

CS-LEX-011. Parsing continues across a newline when the expression or statement is syntactically incomplete, including grouped expressions. No backslash or underscore continuation token is required.

let x = 10; let y = 20
let total =
    subtotal +
    tax

2.5 Numeric and Boolean literals

CS-LEX-012. Integers support decimal digits and 0x, 0b, 0o prefixes. Leading zeros alone do not indicate octal. Decimal-point literals require digits on both sides of the point. Scientific notation requires a complete exponent. Source decimal punctuation is culture-independent.

CS-LEX-013. Ordinary numeric suffixes such as m, f, L and u, digit separators, English magnitude decorators, and nondecimal fractional forms are not accepted. Fixed-duration suffixes are a distinct literal form. Boolean literals are true and false.

CS-TYPE-003. Target typing takes precedence. Without a target, integral inference chooses Integer, then Long, then positive ULong if representable; otherwise compilation reports overflow. Plain fractional notation defaults to Decimal; scientific notation defaults to Double. A default inference choice must not conceal an otherwise ambiguous overload.

Unary sign is an operator; signed boundary values follow numeric range rules.

CS-LEX-014. A numeric literal immediately followed by tick, microsec, ms, sec, min, hour or day denotes a fixed TimeSpan. A tick is 100 ns; a day is exactly 24 hours. No whitespace separates value and unit. Unrepresentable durations are diagnosed. Calendar months/years are not duration suffixes.

2.6 Quoted, raw and multiline strings

CS-LEX-015. Both single and double quotes delimit String. Double the active quote character to include it. Backslash has no general string-escape meaning. A required Char target may accept a literal representing exactly one UTF-16 code unit; the natural type remains String and wins when an overload can use it directly. This does not create a general implicit runtime String-to-Char conversion.

let message = "He said ""Hello"""
let path = "C:\Logs\app.log"
let apostrophe = 'It''s finished'

CS-LEX-016. Multiline strings use matching fences of at least three identical quote characters. Opening and closing fences must have the same quote character and length. A longer fence allows shorter quote runs inside the text.

CS-LEX-017. Ordinary strings recognize {{expression}} interpolation. raw disables interpolation. Doubling the opening delimiter emits a literal {{; }} outside an interpolation is ordinary text. An unclosed interpolation is a compile-time error.

let greeting = "Hello {{customer.Name}}"
let template = raw "Hello {{customer.Name}}"
let escaped = "Use {{{{name}} in the documentation"

The escaped example produces Use {{name}} in the documentation: four opening braces encode two literal opening braces, and the two closing braces remain ordinary text outside an interpolation.

CS-LEX-018. Interpolated expressions evaluate once in source order. nothing contributes empty text. Evaluation exceptions propagate; formatting does not suppress them. Normal formatting uses current culture unless an explicit library API supplies another policy.

CS-LEX-019. Multiline indentation is preserved unless trim is requested. trim trims each line independently. Logical line breaks use the runtime platform newline. raw controls interpolation, not a template engine.

The logical line break immediately after the opening fence and the one immediately before the closing fence are excluded from the value. Exclude only those boundary line breaks, not all leading/trailing blank lines or whitespace. Additional blank content lines at the front or end are preserved. A line break shared by both boundaries is excluded once. The same boundary rule applies to raw and interpolation-enabled multiline strings; trim retains its separate per-line meaning.

let message = """
Hello
World
"""

let padded = """

Hello
World

"""

Reference C# values:

var message = "Hello" + System.Environment.NewLine + "World";
var padded = System.Environment.NewLine + "Hello"
    + System.Environment.NewLine + "World" + System.Environment.NewLine;

The ordinary value starts with H and ends with d. The padded value includes one intentional line break at each end. Neither form uses a whole-string Trim that could remove explicitly authored blank content lines.

let message = trim """
    Customer: {{customer.Name}}
    Balance: {{customer.Balance:N2}}
"""

Character and Unicode library contracts

CS-TEXT-001. Ordinary String storage and iteration use UTF-16; a Char is one code unit, not necessarily a complete Unicode scalar or displayed character. The fixed single-unit Chars values CR, LF, Tab, Null, Escape, Backspace, FormFeed, VerticalTab, Space, SingleQuote, DoubleQuote and Backslash have CLR Char type. Chars.NewLine and Chars.CRLF have String type; NewLine equals Environment.NewLine and CRLF is exactly carriage return followed by line feed. Named Unicode values NBSP, ZeroWidthSpace, BOM, Bullet, Ellipsis, LineSeparator, ParagraphSeparator and ReplacementCharacter have String type. These are ordinary library values, not keywords.

CS-TEXT-002. Chars.FromCodeUnit(value) accepts Long (System.Int64) and returns the Char for a numeric code unit in 0–65535. Ordinary Integer arguments widen to Long. Validate the full input before narrowing. Below zero or above 65535, throw ArgumentOutOfRangeException rather than truncating/wrapping. Surrogate-range units are permitted: constructing one unit does not assert that it is a complete scalar. Chars.CodeUnit(value) returns Integer (System.Int32), the numeric value of its Char without scalar interpretation.

CS-TEXT-003. Unicode.FromCodePoint(value) accepts Long (System.Int64) and returns String text for exactly one valid Unicode scalar. Ordinary Integer arguments widen to Long; validate the full input before narrowing for a native scalar API. Valid scalar values are 0–0x10FFFF except 0xD800–0xDFFF. An invalid input throws ArgumentOutOfRangeException, with no replacement-character fallback. A supplementary scalar is represented by its valid UTF-16 surrogate pair; it is not narrowed to Char. U+0000 and noncharacter scalar values remain valid scalar inputs; these conversions impose no application text/content policy.

CS-TEXT-004. Unicode.CodePoint(text) requires the entire String to contain exactly one valid scalar and returns its numeric value as Integer (System.Int32). Empty, multi-scalar or malformed text throws ArgumentException; nothing throws ArgumentNullException. A valid surrogate pair counts as one scalar. Do not take only the first scalar, ignore trailing text, normalize the input or group several scalars as one merely because they display as one grapheme.

CS-TEXT-005. Unicode.CodePoints(text) traverses complete scalars in source order, combining valid surrogate pairs. When traversal reaches an isolated high or low surrogate, throw ArgumentException rather than emitting U+FFFD or continuing beyond it. Empty text has no scalars; nothing throws ArgumentNullException. This helper does not rewrite stored String data or change ordinary Char iteration. The deferred execution/result contract in CS-TEXT-006 applies alongside this strict decoding rule.

CS-TEXT-006. Unicode.CodePoints returns a deferred streaming IEnumerable <of Integer>. Reject nothing with ArgumentNullException at the call, before returning the sequence; decode only as each enumeration advances. Each enumeration owns a fresh traversal position. A valid prefix may already have been yielded when a later malformed unit raises ArgumentException. Stopping early does not decode an unvisited suffix. Empty text supplies an empty sequence, not nothing. There is no automatic materialization or grapheme grouping; explicit ToList/ToArray requests a snapshot.

let tab as Char = Chars.Tab
let face = Unicode.FromCodePoint(0x1F600)
let faceNumber = Unicode.CodePoint(face)

Face contains 😀 (two UTF-16 code units); faceNumber is 0x1F600. Chars.FromCodeUnit(0xD800) produces a high-surrogate Char, while Unicode.FromCodePoint(0xD800) throws. Unicode.CodePoint("AB") also throws because the String contains two scalars.

2.7 Diagnostics and reference C# lowering

A C# reference pattern for exact single-scalar decoding uses ordinary System.Text.Rune and System.Buffers.OperationStatus:

static int ReadSingleScalar(string text)
{
    ArgumentNullException.ThrowIfNull(text);
    var status = System.Text.Rune.DecodeFromUtf16(text.AsSpan(),
        out var rune, out int consumed);
    if (status != System.Buffers.OperationStatus.Done || consumed != text.Length)
        throw new ArgumentException("Expected exactly one Unicode scalar.", nameof(text));
    return rune.Value;
}

For strict multi-scalar traversal, decode at successive offsets and throw on any status other than Done; only after a successful decode emit the scalar and advance by consumed code units. Native String.EnumerateRunes replaces malformed units and therefore does not alone implement CS-TEXT-005. A scalar-construction pattern validates the full Long against 0–0x10FFFF and excludes surrogate values before narrowing to Integer for new Rune((int)value).ToString(). Checking only a narrowed value can incorrectly accept a large invalid input. Code-unit construction validates the full Long against 0–65535 before conversion to Char. CodePoints wraps strict decoding in a separate iterator so the outer call validates nothing immediately and each iterator stores its own position. Do not put the only null check inside the iterator, which would defer its exception. The numeric input and inverse widths above are fixed; CS-TEXT-007 specifies the public export layout while the following decoding patterns remain private implementation illustrations:

static IEnumerable<int> ReadScalars(string text)
{
    ArgumentNullException.ThrowIfNull(text);
    return Enumerate();

    IEnumerable<int> Enumerate()
    {
        int offset = 0;
        while (offset < text.Length)
        {
            int scalar = DecodeAt(text, offset, out int consumed);
            yield return scalar;
            offset += consumed;
        }
    }
}

static int DecodeAt(string text, int offset, out int consumed)
{
    var status = System.Text.Rune.DecodeFromUtf16(text.AsSpan(offset),
        out var rune, out consumed);
    if (status != System.Buffers.OperationStatus.Done)
        throw new ArgumentException("Malformed UTF-16.", nameof(text));
    return rune.Value;
}

Reject duplicate case-insensitive names, invalid identifier characters, unterminated nested comments/interpolation, malformed numeric forms and independent include compilation. These are rule obligations, not assigned implementation diagnostic codes.

For ordinary CLR numeric targets, the following illustrates CS-TYPE-003:

let amount = 12.50
let scale = 1e3

decimal amount = 12.50m;
double scale = 1e3;

The C# suffix belongs to generated C#. It does not become accepted DirectSharp syntax. Escaped source names may similarly map to C# escaped identifiers after DirectSharp binding; the backend must preserve source identity and diagnostics.

Character helper exports

CS-TEXT-007. DirectSharp.Chars and DirectSharp.Unicode are root-level static classes. Fixed Chars Char values and its CRLF String, plus the eight named Unicode String values in CS-TEXT-001, are public const fields with their stated values. Chars.NewLine is a public static String getter returning Environment.NewLine. FromCodeUnit(Long) returns Char and CodeUnit(Char) returns Integer on Chars; FromCodePoint(Long) returns String, CodePoint(String) returns Integer and CodePoints(String) returns IEnumerable<Integer> on Unicode, all static. Preserve CS-TEXT-002–006's validation, widths, absence and traversal timing. Fixed fields have ordinary CLR constant-context/inlining behavior; runtime platform NewLine is not a portable constant. No replacement mode, grapheme indexing or extra text/numeric overload is selected.

Text transformation profiles

These are ordinary library operations. Their profiles do not change identifier identity, normal String comparison or UTF-16 storage. Export containers and additional overloads are separate from these selected behaviors.

CS-TEXT-008. Upper and Lower use String.ToUpperInvariant and String.ToLowerInvariant semantics. A missing source throws ArgumentNullException; empty remains empty. Do not trim, normalize, tokenize or impose separate whole-String scalar validation. Native .NET casing and runtime Unicode tables apply; culture-independent behavior does not promise unchanged Unicode tables across runtime versions. Explicit culture-sensitive presentation can use ordinary .NET APIs.

CS-TEXT-009. UrlEncode encodes one URI component using Uri.EscapeDataString semantics after rejecting a missing source with ArgumentNullException. Empty remains empty. Treat all input as data; do not parse a full URL, preserve its separators as syntax or decode existing percent sequences. Native escaping and invalid-input behavior apply. Space becomes percent20, slash is encoded and an existing percent is encoded again. No decoder, form encoding or implicit URL builder is introduced.

CS-TEXT-010. JsonEscape and CSharpEscape return escaped String contents without surrounding double quotes. Missing source throws ArgumentNullException and empty produces empty contents. Required quote/backslash/control escaping must produce valid contents for the selected target; these operations are not object serialization or source-expression evaluation. This rule selects the delimiter/source-null convention, not a complete Unicode, malformed-UTF-16, HTML-embedding or canonical escape-spelling profile. Those additional policies are not inferred from the helper names.

CS-TEXT-011. CamelCase, PascalCase, SnakeCase and KebabCase use shared segmentation. Non-letter/digit/combining-mark text separates words, with no empty words for repeated separators. Combining marks stay with their preceding word; a leading unattached mark is a separator. Split lowercase-or-digit-to-uppercase transitions and split before the last uppercase letter of an uppercase run followed by lowercase text. Digits stay attached to the preceding word; a following uppercase transition starts another word. Use runtime Unicode letter/digit/category tests and invariant simple casing. Strictly decode complete scalars, throwing ArgumentException for malformed UTF-16; missing source throws ArgumentNullException, and empty/separator-only source produces empty String. Do not normalize or transliterate. Snake/Kebab lowercase words and join with underscore/hyphen. Pascal uses initial-uppercase/rest-lowercase words; Camel lowercases the first word and applies Pascal treatment to later words. Initial capitalization skips any combining mark already belonging to its base. HTTPServer2ID therefore becomes http_server2_id under SnakeCase and httpServer2Id under CamelCase. The resulting text is not guaranteed to be a legal source identifier or locale-sensitive natural-language title.

Text layout and escaped contents

These profiles supplement the delimiter convention in CS-TEXT-010. Ordinary String storage/indexing is unchanged; CS-TEXT-017 below specifies their public exports and Integer layout inputs.

CS-TEXT-012. Wrap width and Truncate/Ellipsis length count Unicode text elements recognized by the target runtime's System.Globalization.StringInfo segmentation. Retain complete recognized elements and original code units, with no normalization/transliteration. Runtime Unicode segmentation may evolve. This measures neither rendered columns nor font/glyph width and does not change ordinary UTF-16 indexing or iteration.

CS-TEXT-013. Truncate/Ellipsis length is a total budget including the suffix. If source length is within budget, return it unchanged without adding a suffix. Otherwise reserve suffix space and retain a fitting source prefix. If suffix alone exceeds the budget, return its fitting prefix with no source prefix. Zero length returns empty text. Ellipsis uses Unicode.Ellipsis as suffix; plain Truncate uses empty suffix. Count each source/suffix using CS-TEXT-012. This is a text-length budget, not a byte-length or rendered-column guarantee.

CS-TEXT-014. Wrap preserves existing CRLF, CR and LF boundaries exactly, including empty lines and a final boundary. Within each line, words are maximal runs separated by ASCII space/tab; collapse separator runs to one ASCII space between words and omit leading/trailing separators. Greedily pack complete words up to width; split oversized words into pieces of at most width using complete text elements, then continue packing after a final short piece. Newly inserted boundaries use Environment.NewLine. Empty source remains empty. Other whitespace, including NBSP, is ordinary word content. No locale-sensitive line breaking, hyphenation, markup preservation or terminal tab expansion is implied.

CS-TEXT-015. Layout helpers reject missing source with ArgumentNullException first. Then reject negative Truncate/Ellipsis length or nonpositive Wrap width with ArgumentOutOfRangeException; a supplied missing suffix throws ArgumentNullException after length validation. Empty suffix and zero truncation length are valid. Validate complete source and supplied suffix UTF-16 before producing any successful result; an isolated surrogate throws ArgumentException even in a discarded tail, unused suffix or zero-length request. No replacement, successful partial result, numeric clamping or new Try helper is introduced. These local rules do not alter imported APIs or ordinary String validity.

CS-TEXT-016. JsonEscape/CSharpEscape reject missing source with ArgumentNullException and validate complete well-formed UTF-16 before returning; an isolated surrogate throws ArgumentException. Escape double quote and backslash; use short escapes for backspace, form feed, line feed, carriage return and tab. Encode other U+0000–U+001F, all U+007F–U+009F and U+2028/U+2029 with backslash-u plus four uppercase hexadecimal digits. Preserve other valid Unicode scalars, including supplementary surrogate pairs, solidus and single quote, without normalization. Return unquoted contents under CS-TEXT-010, with empty input producing empty contents. Targets are JSON double-quoted strings and ordinary C# double-quoted strings, not C# verbatim/raw/interpolated forms, JavaScript expressions or HTML-safe script embedding. ASCII markup remains unescaped; object serialization and HTML/XML encoding retain their separate contexts.

Text helper exports and markup profiles

CS-TEXT-017. The public static class DirectSharp.Text.TextExtensions exports the fifteen String-returning extension members in the API catalogue: Upper, Lower, CamelCase, PascalCase, SnakeCase, KebabCase, UrlEncode, HtmlEncode, XmlEncode, JsonEscape, CSharpEscape, Wrap(width), Ellipsis(length), Truncate(length) and Truncate(length, suffix). Each receives a String source; suffix is String, and width/length are Integer/System.Int32. Preserve the corresponding profiles and validation order. No Long overload or implicit narrowing is introduced. The child namespace requires explicit import; automatic root DirectSharp import does not import it. Ordinary extension lookup/ambiguity applies, with explicit static qualification available and no hidden helper priority. These declarations select neither package/assembly policy nor extra overloads.

CS-TEXT-018. HtmlEncode rejects missing source with ArgumentNullException, then uses System.Net.WebUtility.HtmlEncode, preserving native Unicode/escape behavior and empty input. The context is HTML text or quoted HTML attribute data, not JavaScript, CSS or URI data. Do not detect/decode existing entities or add normalization/strict scalar validation beyond that native profile.

CS-TEXT-019. XmlEncode rejects missing source with ArgumentNullException, then buffers XmlWriter.WriteString with ConformanceLevel.Fragment, OmitXmlDeclaration=true, CheckCharacters=true and NewLineHandling.Entitize. Return only the encoded XML element-text fragment; empty input returns empty text. Preserve native XML character/UTF-16 validation failures, including ArgumentException for invalid XML characters or isolated surrogates. Entitization preserves carriage returns through an XML reader. Do not detect/decode existing entities or treat the result as an attribute encoder, tag/name or complete document. XML structural and attribute construction use the writer's corresponding operations.

Specification contents · Previous: 1. Scope and conformance · Next: 3. Names, namespaces and imports