HomePage | QuickStart | Navigate

The Set|File Site

Page generated from: SET_File_ImplementationGuide_v4_2.md


SET File Implementation Guide

Version 4.2
Updated: December 2025

---

About This Guide

This document provides comprehensive guidance for implementing SET file parsers, query languages, and extended functionality. These are recommendations and patterns for building SET-based systems, not requirements of the core specification.

For the specifications, see:

SET File Format One-Page Quick-Start

SET File Format Core Specification v4.2

SET File Format Full Specification v4.2

SET File Format QSet Specification v4.2

---

Table of Contents

---

1. Query Language (SetQL)

SetQL provides a simple query syntax for filtering and selecting data from SET files. This is an optional feature - implementations may choose whether to support it.

1.1 Basic Syntax


FROM [GroupName] WHERE field=value
FROM [GroupName] SELECT field1,field2
FROM [GroupName] WHERE field>value ORDER BY field

1.2 Supported Operations

Comparison Operators:

Pattern Matching: List Operations: Logical Operators:

1.3 Query Components

FROM clause - Specifies the group to query 

FROM [USERS]
FROM [DATABASE_CONFIG]
WHERE clause - Filters records 

WHERE role='admin'
WHERE age>18 AND status='active'
WHERE email LIKE '%@example.com'
SELECT clause - Chooses specific fields 

SELECT username,email
SELECT *
ORDER BY clause - Sorts results 

ORDER BY last_name
ORDER BY age DESC

1.4 Examples

Simple queries: 

FROM [USERS] WHERE role='admin'
FROM [PRODUCTS] WHERE price>100
FROM [SETTINGS] WHERE key LIKE 'Email%'
Complex queries: 

FROM [EMPLOYEES] WHERE department='Engineering' AND salary>75000 ORDER BY hire_date
FROM [ORDERS] WHERE status IN ('pending','processing') AND total>500
FROM [CONTACTS] WHERE (city='Seattle' OR city='Portland') AND active=true
With SELECT: 

FROM [USERS] SELECT username,email WHERE role='user'
FROM [PRODUCTS] SELECT name,price WHERE category='Electronics' ORDER BY price DESC

1.5 Text Block References

Text blocks are not directly queryable, but references are resolved in query results:


FROM [ARTICLES] SELECT title,body

If the body field contains [{ARTICLE_1_BODY}], the query result includes the resolved text content.

1.6 Implementation Notes

---

2. Implementation Patterns & Conventions

The SET file format's simple structure enables many powerful patterns through creative use of existing features. These are conventions, not requirements - implementations choose what makes sense for their use cases.

2.1 Hierarchical Data via Dot Notation

Use dots in key names to represent nested structures.


[DATABASE_CONFIG]
host|localhost
port|5432
connection.pool.min|5
connection.pool.max|20
connection.timeout|30
ssl.enabled|true
ssl.cert|/path/to/cert.pem
[EOG]
Simple parser: Treats connection.pool.min as a single key
Advanced parser: Builds nested structure: 
javascript
{
  host: "localhost",
  connection: {
    pool: { min: 5, max: 20 },
    timeout: 30
  },
  ssl: {
    enabled: true,
    cert: "/path/to/cert.pem"
  }
}

2.2 Runtime Calculation Fields (Implementation Pattern)

Some implementations may choose to support calculated fields that are generated at parse time rather than stored in the file.

Convention: Use :: prefix to mark calculated fields in field definitions.

Example: 

[SALES]
{date|product|amount|::tax|::total}
2025-01-01|Widget A|100.00
2025-01-02|Widget B|150.00
2025-01-03|Widget C|200.00
Parser behavior: When the parser encounters ::tax and ::total, it:
    • Calculates tax (e.g., amount 0.08)
    • Calculates total (e.g., amount + tax)
    • Adds these fields to the returned data structure
Returned data might look like: 
javascript
[
  {date: "2025-01-01", product: "Widget A", amount: "100.00", tax: "8.00", total: "108.00"},
  {date: "2025-01-02", product: "Widget B", amount: "150.00", tax: "12.00", total: "162.00"},
  {date: "2025-01-03", product: "Widget C", amount: "200.00", tax: "16.00", total: "216.00"}
]

Notes:

  • This is not part of the SET file format specification
  • It's a convention some implementations may choose to support
  • The calculation logic is entirely implementation-specific
  • The :: prefix is just a convention to distinguish calculated from stored fields
  • Simple parsers can ignore :: fields or treat them as documentation
Use cases:
  • Computed totals, subtotals, running totals
  • Calculated dates (e.g., expiration date from creation date)
  • Derived values (e.g., full name from first + last name)
  • Display formatting (e.g., formatted currency from raw numbers)

2.3 Type Hints via Key Conventions

Add type information through naming conventions.

Suffix notation: 

[SETTINGS]
maxUsers_int|50
timeout_float|30.5
debugMode_bool|true
database_null|
tags_array|development,testing,production
[EOG]
Colon notation: 

[SETTINGS]
maxUsers:int|50
timeout:float|30.5
debugMode:bool|true
[EOG]
Value prefixes: 

[SETTINGS]
maxUsers|i:50
timeout|f:30.5
debugMode|b:true
[EOG]

Choose whatever convention fits your implementation.

2.4 Arrays and Lists

Horizontal arrays (positional fields): 

[COLORS]
{name|red|green|blue}
Primary Red|255|0|0
Sky Blue|135|206|235
Forest Green|34|139|34
[EOG]
Vertical arrays (repeated keys): 

[ALLOWED_IPS]
ip|192.168.1.1
ip|192.168.1.2
ip|192.168.1.3
ip|10.0.0.5
[EOG]
Comma-separated lists in values: 

[USER_ROLES]
admin|create,read,update,delete
editor|read,update
viewer|read
[EOG]
Nested arrays using secondary delimiter: 

[PERMISSIONS]
{role|permissions}
admin|users:create!users:delete!posts:all
editor|posts:create!posts:edit!posts:delete
viewer|posts:read!comments:read
[EOG]

The secondary delimiter (default !) allows nested array structures within a single field.

2.5 Version Suffixes

Use version numbers in group names for managing configuration versions:


[DATABASE_V1]
host|localhost
port|3306
[EOG]

[DATABASE_V2]
host|db.example.com
port|5432
pool_size|20
[EOG]

2.6 Environment-Specific Configurations


[DATABASE_PRODUCTION]
host|prod.db.example.com
port|5432
[EOG]

[DATABASE_STAGING]
host|staging.db.example.com
port|5432
[EOG]

[DATABASE_DEVELOPMENT]
host|localhost
port|5432
[EOG]

2.7 Base64 Encoding for Binary Data

Store binary data as Base64-encoded text in text blocks:


[APP_CONFIG]
icon|[{APP_ICON_BASE64}]
certificate|[{SSL_CERT_BASE64}]
[EOG]

[{APP_ICON_BASE64}]
iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==
[EOG]

[{SSL_CERT_BASE64}]
MIIDXTCCAkWgAwIBAgIJAKL0UG+mRfQNMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX
[EOG]

---

3. SetTag Extensions

SetTags allow embedding SET file content within other file formats using their native comment syntax.

3.1 Basic SetTag Syntax

Pattern: 

<comment-start> {SETTAG:TagName}
<comment> SET file content here
<comment> {/SETTAG/} <comment-end>

3.2 Language-Specific Examples

JavaScript/PHP: 
javascript
// {SETTAG:AppConfig}
// [SETTINGS]
// AppName|MyApp
// Version|1.0
// {/SETTAG/}
Python: 
python
# {SETTAG:Config}
# [DATABASE]
# Host|localhost
# Port|5432
# {/SETTAG/}
HTML/Markdown: 
html
<!-- {SETTAG:PageConfig}
[META]
Title|My Page
Author|John Doe
{/SETTAG/} -->
CSS: 
css
/* {SETTAG:ThemeConfig}
[COLORS]
Primary|#007bff
Secondary|#6c757d
{/SETTAG/} */
Rust: 
rust
// {SETTAG:BuildConfig}
// [FEATURES]
// Debug|true
// Optimize|false
// {/SETTAG/}

3.3 Extracting SetTags

Implementations can extract SetTags by:

    • Reading the file line by line
    • Looking for {SETTAG:Name} markers
    • Collecting lines until {/SETTAG/} is found
    • Stripping comment characters and spaces
    • Parsing the extracted content as a SET file
Example extraction function (pseudocode): 
javascript
function extractSetTag(filename, tagName) {
  let lines = readFile(filename);
  let inTag = false;
  let content = [];
  
  for (let line of lines) {
    if (line.includes(`{SETTAG:${tagName}}`)) {
      inTag = true;
      continue;
    }
    if (line.includes('{/SETTAG/}')) {
      break;
    }
    if (inTag) {
      // Strip comment markers and leading space
      let cleaned = stripComments(line);
      content.push(cleaned);
    }
  }
  
  return parseSetContent(content.join('\n'));
}

3.4 Use Cases for SetTags

  • Configuration in source files: Embed config data directly in application code
  • Metadata in documents: Store structured metadata in markdown or HTML files
  • Build settings: Include build configuration in the files they affect
  • Documentation: Keep related configuration with its documentation
  • Version control: Track configuration changes alongside code changes
---

4. CRUD Operations

This section provides guidance for implementing Create, Read, Update, and Delete operations on SET files.

4.1 Reading Data

Read entire file: 
javascript
function readSetFile(filename) {
  let content = readFile(filename);
  return parseSetFile(content);
}
Read specific group: 
javascript
function readGroup(filename, groupName) {
  let data = readSetFile(filename);
  return data.groups[groupName];
}
Read specific key-value: 
javascript
function readValue(filename, groupName, key) {
  let group = readGroup(filename, groupName);
  return group[key];
}
Read with text block resolution: 
javascript
function readWithReferences(filename, groupName, key) {
  let value = readValue(filename, groupName, key);
  
  // Check if value is a text block reference
  if (value.match(/^\[{.*}\]$/)) {
    let blockName = value.substring(2, value.length - 2);
    return readGroup(filename, blockName);
  }
  
  return value;
}

4.2 Creating Data

Create new SET file: 
javascript
function createSetFile(filename, config = {}) {
  let content = [];
  
  // Add filename identifier
  content.push(filename);
  content.push('');
  
  // Add [THIS-FILE] if config provided
  if (Object.keys(config).length > 0) {
    content.push('[THIS-FILE]');
    for (let [key, value] of Object.entries(config)) {
      content.push(`${key}|${value}`);
    }
    content.push('[EOG]');
    content.push('');
  }
  
  writeFile(filename, content.join('\n'));
}
Add new group: 
javascript
function addGroup(filename, groupName, data, isTextBlock = false) {
  let content = readFile(filename);
  
  // Add group header
  if (isTextBlock) {
    content += `\n[{${groupName}}]\n`;
    content += data + '\n';
  } else {
    content += `\n[${groupName}]\n`;
    for (let [key, value] of Object.entries(data)) {
      content += `${key}|${value}\n`;
    }
  }
  
  content += '[EOG]\n';
  writeFile(filename, content);
}
Add line to existing group: 
javascript
function addLine(filename, groupName, key, value) {
  let lines = readFileLines(filename);
  let inGroup = false;
  let insertIndex = -1;
  
  for (let i = 0; i < lines.length; i++) {
    if (lines[i].trim() === `[${groupName}]`) {
      inGroup = true;
      continue;
    }
    
    if (inGroup && (lines[i].trim() === '' || 
                     lines[i].trim().startsWith('[EOG]') ||
                     lines[i].trim().startsWith('['))) {
      insertIndex = i;
      break;
    }
  }
  
  if (insertIndex > -1) {
    lines.splice(insertIndex, 0, `${key}|${value}`);
    writeFileLines(filename, lines);
  }
}

4.3 Updating Data

Update existing value: 
javascript
function updateValue(filename, groupName, key, newValue) {
  let lines = readFileLines(filename);
  let inGroup = false;
  
  for (let i = 0; i < lines.length; i++) {
    if (lines[i].trim() === `[${groupName}]`) {
      inGroup = true;
      continue;
    }
    
    if (inGroup) {
      if (lines[i].trim() === '' || 
          lines[i].trim().startsWith('[EOG]') ||
          lines[i].trim().startsWith('[')) {
        break;
      }
      
      let parts = lines[i].split('|');
      if (parts[0].trim() === key) {
        parts[1] = newValue;
        lines[i] = parts.join('|');
        writeFileLines(filename, lines);
        return true;
      }
    }
  }
  
  return false; // Key not found
}
Update entire group: 
javascript
function updateGroup(filename, groupName, newData) {
  let lines = readFileLines(filename);
  let startIndex = -1;
  let endIndex = -1;
  
  for (let i = 0; i < lines.length; i++) {
    if (lines[i].trim() === `[${groupName}]`) {
      startIndex = i;
      continue;
    }
    
    if (startIndex > -1 && (lines[i].trim() === '' || 
                             lines[i].trim().startsWith('[EOG]') ||
                             lines[i].trim().startsWith('['))) {
      endIndex = i;
      break;
    }
  }
  
  if (startIndex > -1) {
    let newLines = [`[${groupName}]`];
    for (let [key, value] of Object.entries(newData)) {
      newLines.push(`${key}|${value}`);
    }
    newLines.push('[EOG]');
    
    lines.splice(startIndex, endIndex - startIndex, ...newLines);
    writeFileLines(filename, lines);
    return true;
  }
  
  return false;
}

4.4 Deleting Data

Delete specific key: 
javascript
function deleteKey(filename, groupName, key) {
  let lines = readFileLines(filename);
  let inGroup = false;
  
  for (let i = 0; i < lines.length; i++) {
    if (lines[i].trim() === `[${groupName}]`) {
      inGroup = true;
      continue;
    }
    
    if (inGroup) {
      if (lines[i].trim() === '' || 
          lines[i].trim().startsWith('[EOG]') ||
          lines[i].trim().startsWith('[')) {
        break;
      }
      
      let parts = lines[i].split('|');
      if (parts[0].trim() === key) {
        lines.splice(i, 1);
        writeFileLines(filename, lines);
        return true;
      }
    }
  }
  
  return false;
}
Delete entire group: 
javascript
function deleteGroup(filename, groupName) {
  let lines = readFileLines(filename);
  let startIndex = -1;
  let endIndex = -1;
  
  for (let i = 0; i < lines.length; i++) {
    if (lines[i].trim() === `[${groupName}]` || 
        lines[i].trim() === `[{${groupName}}]`) {
      startIndex = i;
      continue;
    }
    
    if (startIndex > -1 && (lines[i].trim() === '' || 
                             lines[i].trim().startsWith('[EOG]') ||
                             (lines[i].trim().startsWith('[') && i > startIndex))) {
      endIndex = i + 1; // Include [EOG] or next group
      break;
    }
  }
  
  if (startIndex > -1) {
    lines.splice(startIndex, endIndex - startIndex);
    writeFileLines(filename, lines);
    return true;
  }
  
  return false;
}

4.5 Transaction Safety

For critical operations, implement atomic writes:

javascript
function atomicUpdate(filename, updateFunction) {
  let tempFile = filename + '.tmp';
  let backupFile = filename + '.backup';
  
  try {
    // Create backup
    copyFile(filename, backupFile);
    
    // Perform update on temp file
    copyFile(filename, tempFile);
    updateFunction(tempFile);
    
    // Verify temp file is valid
    if (validateSetFile(tempFile)) {
      // Replace original with temp
      moveFile(tempFile, filename);
      deleteFile(backupFile);
      return true;
    } else {
      throw new Error('Validation failed');
    }
  } catch (error) {
    // Restore from backup
    if (fileExists(backupFile)) {
      copyFile(backupFile, filename);
    }
    deleteFile(tempFile);
    return false;
  }
}

---

5. Validation & Error Handling

5.1 File-Level Validation

Basic structure validation: 
javascript
function validateSetFile(content) {
  let errors = [];
  let lines = content.split('\n');
  let inGroup = false;
  let inTextGroup = false;
  let currentGroup = null;
  let groupNames = new Set();
  
  for (let i = 0; i < lines.length; i++) {
    let line = lines[i];
    let trimmed = line.trim();
    
    // Check for group start
    if (trimmed.match(/^\[([A-Za-z0-9_-]+)\]$/)) {
      let groupName = trimmed.substring(1, trimmed.length - 1);
      
      // Check for duplicate group names
      if (groupNames.has(groupName)) {
        errors.push(`Line ${i + 1}: Duplicate group name '${groupName}'`);
      }
      groupNames.add(groupName);
      
      inGroup = true;
      inTextGroup = false;
      currentGroup = groupName;
      continue;
    }
    
    // Check for text group start
    if (trimmed.match(/^\[\{([A-Za-z0-9_-]+)\}\]$/)) {
      let groupName = trimmed.substring(2, trimmed.length - 2);
      
      if (groupNames.has(groupName)) {
        errors.push(`Line ${i + 1}: Duplicate group name '${groupName}'`);
      }
      groupNames.add(groupName);
      
      inGroup = true;
      inTextGroup = true;
      currentGroup = groupName;
      continue;
    }
    
    // Check for invalid group names
    if (trimmed.startsWith('[') && !trimmed.match(/^\[[A-Za-z0-9_-]+\]$/) && 
        !trimmed.match(/^\[\{[A-Za-z0-9_-]+\}\]$/) &&
        trimmed !== '[EOG]' && trimmed !== '[EOF]') {
      errors.push(`Line ${i + 1}: Invalid group name format '${trimmed}'`);
    }
    
    // Check for [EOG] or empty line ending groups
    if ((trimmed === '[EOG]' || trimmed === '') && !inTextGroup) {
      inGroup = false;
      currentGroup = null;
    }
  }
  
  return {
    valid: errors.length === 0,
    errors: errors
  };
}

5.2 Group-Level Validation

Validate group structure: 
javascript
function validateGroup(groupContent, groupType = 'data') {
  let errors = [];
  let lines = groupContent.split('\n').filter(l => l.trim());
  
  if (groupType === 'keyvalue') {
    for (let i = 0; i < lines.length; i++) {
      if (!lines[i].includes('|')) {
        errors.push(`Line ${i + 1}: Key-value line must contain delimiter '|'`);
      }
      
      let parts = lines[i].split('|');
      if (parts.length < 2) {
        errors.push(`Line ${i + 1}: Key-value line must have at least key and value`);
      }
      
      if (parts[0].trim() === '') {
        errors.push(`Line ${i + 1}: Key cannot be empty`);
      }
    }
  }
  
  if (groupType === 'table') {
    // First line should be field definition
    if (lines.length > 0 && !lines[0].startsWith('{')) {
      errors.push('Table group must start with field definition {field1|field2|...}');
    }
    
    if (lines.length > 1) {
      let fieldCount = lines[0].split('|').length;
      for (let i = 1; i < lines.length; i++) {
        let dataCount = lines[i].split('|').length;
        if (dataCount > fieldCount && !lines[i].includes(':::') && !lines[i].includes('…')) {
          errors.push(`Line ${i + 1}: Data field count exceeds definition (${dataCount} > ${fieldCount})`);
        }
      }
    }
  }
  
  return {
    valid: errors.length === 0,
    errors: errors
  };
}

5.3 Data Type Validation

Validate data types (when using type conventions): 
javascript
function validateTypes(data, schema) {
  let errors = [];
  
  for (let [key, expectedType] of Object.entries(schema)) {
    let value = data[key];
    
    switch(expectedType) {
      case 'int':
        if (!Number.isInteger(Number(value))) {
          errors.push(`${key}: Expected integer, got '${value}'`);
        }
        break;
        
      case 'float':
        if (isNaN(parseFloat(value))) {
          errors.push(`${key}: Expected float, got '${value}'`);
        }
        break;
        
      case 'bool':
        if (value !== 'true' && value !== 'false') {
          errors.push(`${key}: Expected boolean (true/false), got '${value}'`);
        }
        break;
        
      case 'email':
        if (!value.match(/^[^\s@]+@[^\s@]+\.[^\s@]+$/)) {
          errors.push(`${key}: Invalid email format '${value}'`);
        }
        break;
        
      case 'url':
        try {
          new URL(value);
        } catch {
          errors.push(`${key}: Invalid URL format '${value}'`);
        }
        break;
    }
  }
  
  return {
    valid: errors.length === 0,
    errors: errors
  };
}

5.4 Reference Validation

Validate text block references: 
javascript
function validateReferences(data) {
  let errors = [];
  let availableBlocks = new Set();
  
  // Collect all text block names
  for (let groupName of Object.keys(data.groups)) {
    if (groupName.startsWith('{') && groupName.endsWith('}')) {
      availableBlocks.add(groupName);
    }
  }
  
  // Check all references
  for (let [groupName, groupData] of Object.entries(data.groups)) {
    if (typeof groupData === 'object') {
      for (let [key, value] of Object.entries(groupData)) {
        if (typeof value === 'string' && value.match(/^\[\{.*\}\]$/)) {
          let refName = value.substring(2, value.length - 2);
          if (!availableBlocks.has(`{${refName}}`)) {
            errors.push(`${groupName}.${key}: Reference to undefined text block '${refName}'`);
          }
        }
      }
    }
  }
  
  return {
    valid: errors.length === 0,
    errors: errors
  };
}

5.5 Error Recovery Strategies

Lenient parsing with error collection: 
javascript
function parseWithRecovery(content) {
  let result = {
    data: {},
    errors: [],
    warnings: []
  };
  
  try {
    result.data = parseSetFile(content);
  } catch (error) {
    result.errors.push(`Fatal parse error: ${error.message}`);
    
    // Attempt partial recovery
    try {
      result.data = parsePartialSetFile(content);
      result.warnings.push('File parsed with errors, some data may be missing');
    } catch {
      result.errors.push('Unable to recover any data from file');
    }
  }
  
  // Validate what we got
  let validation = validateSetFile(content);
  result.errors.push(...validation.errors);
  
  return result;
}
Graceful degradation: 
javascript
function readValueSafe(filename, groupName, key, defaultValue = null) {
  try {
    return readValue(filename, groupName, key);
  } catch (error) {
    console.warn(`Failed to read ${groupName}.${key}: ${error.message}`);
    return defaultValue;
  }
}

---

6. Programming Interface Guidelines

6.1 Parser Design Principles

Minimal parser (Q-Set approach):

  • Parse only sections 1-3 of specification
  • Built-in parsing, no external libraries
  • Single file implementation
  • Focus on read-only operations
Standard parser:
  • Support sections 1-4 of specification
  • Include CRUD operations
  • Basic validation
  • May use helper libraries
Full-featured parser:
  • Complete specification support
  • SetQL query language
  • Advanced validation
  • Transaction safety
  • Reference resolution
  • Optional features (runtime calculations, type hints)

6.2 API Design Patterns

Object-oriented approach: 
javascript
class SetFile {
  constructor(filename) {
    this.filename = filename;
    this.data = this.load();
  }
  
  load() {
    let content = readFile(this.filename);
    return parseSetFile(content);
  }
  
  save() {
    let content = serializeSetFile(this.data);
    writeFile(this.filename, content);
  }
  
  getGroup(name) {
    return this.data.groups[name];
  }
  
  getValue(group, key) {
    return this.data.groups[group]?.[key];
  }
  
  setValue(group, key, value) {
    if (!this.data.groups[group]) {
      this.data.groups[group] = {};
    }
    this.data.groups[group][key] = value;
  }
  
  query(setql) {
    return executeQuery(this.data, setql);
  }
}

// Usage
let config = new SetFile('app.set');
let dbHost = config.getValue('DATABASE', 'Host');
config.setValue('DATABASE', 'Port', '5432');
config.save();
Functional approach: 
javascript
// Pure functions, immutable data
const SetFile = {
  parse: (content) => parseSetFile(content),
  
  serialize: (data) => serializeSetFile(data),
  
  getGroup: (data, groupName) => 
    data.groups[groupName],
  
  getValue: (data, groupName, key) =>
    data.groups[groupName]?.[key],
  
  setValue: (data, groupName, key, value) => ({
    ...data,
    groups: {
      ...data.groups,
      [groupName]: {
        ...data.groups[groupName],
        [key]: value
      }
    }
  }),
  
  query: (data, setql) => executeQuery(data, setql)
};

// Usage
let content = readFile('app.set');
let data = SetFile.parse(content);
let dbHost = SetFile.getValue(data, 'DATABASE', 'Host');
let updated = SetFile.setValue(data, 'DATABASE', 'Port', '5432');
writeFile('app.set', SetFile.serialize(updated));

6.3 Caching Strategies

Lazy loading with cache: 
javascript
class CachedSetFile {
  constructor(filename) {
    this.filename = filename;
    this.cache = null;
    this.lastModified = null;
  }
  
  getData() {
    let currentModTime = getFileModTime(this.filename);
    
    if (this.cache === null || currentModTime > this.lastModified) {
      this.cache = this.load();
      this.lastModified = currentModTime;
    }
    
    return this.cache;
  }
  
  invalidate() {
    this.cache = null;
  }
  
  getValue(group, key) {
    return this.getData().groups[group]?.[key];
  }
}

6.4 Stream Processing

For large files: 
javascript
function* streamSetGroups(filename) {
  let lines = readFileStream(filename);
  let currentGroup = null;
  let groupLines = [];
  
  for (let line of lines) {
    let trimmed = line.trim();
    
    if (trimmed.match(/^\[[A-Za-z0-9_-]+\]$/)) {
      // New group starting
      if (currentGroup) {
        yield {
          name: currentGroup,
          content: parseGroupLines(groupLines)
        };
      }
      currentGroup = trimmed.substring(1, trimmed.length - 1);
      groupLines = [];
    } else if (trimmed === '' || trimmed === '[EOG]') {
      // Group ending
      if (currentGroup) {
        yield {
          name: currentGroup,
          content: parseGroupLines(groupLines)
        };
        currentGroup = null;
        groupLines = [];
      }
    } else if (currentGroup) {
      groupLines.push(line);
    }
  }
  
  // Handle last group if file doesn't end with EOG
  if (currentGroup && groupLines.length > 0) {
    yield {
      name: currentGroup,
      content: parseGroupLines(groupLines)
    };
  }
}

// Usage
for (let group of streamSetGroups('large.set')) {
  console.log(`Processing ${group.name}...`);
  processGroup(group.content);
}

6.5 Internationalization Support

Handling locale settings from [THIS-FILE]: 
javascript
function parseWithLocale(content) {
  let data = parseSetFile(content);
  let locale = data.groups['THIS-FILE']?.Localize;
  
  if (locale) {
    let [normalization, language, direction] = locale.split('|');
    
    // Apply normalization
    if (normalization === 'NFC') {
      data = normalizeNFC(data);
    }
    
    // Set text direction
    data.metadata = {
      ...data.metadata,
      textDirection: direction || 'LTR',
      language: language || 'en-US'
    };
  }
  
  return data;
}

---

7. Complete Examples

7.1 Simple Configuration File


app_config.set

[THIS-FILE]
Version|4.2
Created|2025-12-04
[EOG]

[DATABASE]
Host|localhost
Port|5432
Database|myapp
User|dbuser
Password|[{DB_PASSWORD}]
[EOG]

[APP_SETTINGS]
AppName|My Application
Version|1.0.0
Debug|false
LogLevel|info
[EOG]

[EMAIL]
SMTPHost|smtp.example.com
SMTPPort|587
FromAddress|noreply@example.com
[EOG]

[{DB_PASSWORD}]
encrypted_password_here_base64_encoded
[EOG]

[EOF]

7.2 Data Table with Field Definitions


users.set

[THIS-FILE]
Version|4.2
Delimiters|:[]:{}:|:\:…:!
[EOG]

[USERS]
{id|username|email|role|created_at}
1|alice|alice@example.com|admin|2025-01-15
2|bob|bob@example.com|editor|2025-02-20
3|charlie|charlie@example.com|viewer|2025-03-10
4|diana|diana@example.com|editor|2025-03-15
[EOG]

[USER_PERMISSIONS]
{user_id|resource|actions}
1|all|create!read!update!delete
2|posts|create!read!update!delete
2|comments|read!update
3|posts|read
3|comments|read
4|posts|create!read!update!delete
[EOG]

[EOF]

7.3 Mixed Content with Text Blocks


project.set

[THIS-FILE]
Version|4.2
Created|2025-12-04
[EOG]

[PROJECT_INFO]
Name|Advanced Demo
Version|2.0
Description|[{PROJECT_DESC}]
License|[{LICENSE}]
[EOG]

[DEPENDENCIES]
{package|version|source}
react|18.2.0|npm
typescript|5.0.0|npm
vite|4.3.0|npm
[EOG]

[BUILD_CONFIG]
Target|es2020
OutDir|dist
SourceMap|true
Minify|false
[EOG]

[{PROJECT_DESC}]
This is a demonstration project showcasing the SET file format v4.2.

It includes multiple types of data:
- Configuration settings
- Tabular data with field definitions
- Text blocks for longer content
- Nested arrays using secondary delimiters

The project serves as both documentation and a working example.
[EOG]

[{LICENSE}]
MIT License

Copyright (c) 2025 Example Corp

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
[EOG]

[EOF]

7.4 Advanced Features Demo


advanced_demo.set

[THIS-FILE]
Version|4.2
Delimiters|:[]:{}:|:\:…:!
Created|2025-12-04
[EOG]

[SALES_DATA]
{date|product|quantity|price|::subtotal|::tax|::total}
2025-01-15|Widget A|10|25.00
2025-01-16|Widget B|5|50.00|:::rush_order:true
2025-01-17|Widget C|3|100.00
[EOG]

[CONTACT_INFO]
{id|name|phone|email|address|city|state|zip}
1|Alice Smith|555-1234|alice@example.com|123 Main St|Seattle|WA|98101
2|Bob Jones|555-5678|bob@example.com|…
3|Carol White|555-9012|carol@example.com|456 Oak Ave|Portland|OR|97201
[EOG]

[API_ENDPOINTS]
{name|url|method}
GetUsers|https://api.example.com/users|GET
:!CreateUser!https://api.example.com/users?format=json|include=profile|expand=roles!POST
UpdateUser|https://api.example.com/users/{id}|PUT
DeleteUser|https://api.example.com/users/{id}|DELETE
[EOG]

[NESTED_PERMISSIONS]
{role|modules}
admin|users!posts!comments!settings
editor|posts!comments
author|posts
moderator|comments!users
[EOG]

[README]
[{PROJECT_README}]
[EOG]

[{PROJECT_README}]
# Advanced Demo Project

## Features Demonstrated

1. **Runtime Calculation Pattern (::)**
   - Implementation-specific feature (see Section 2.2)
   - Calculated subtotal, tax, and total fields
   - Not part of core spec, but a common convention

2. **Single-Use Fields (:::)**
   - Per-record metadata without modifying field definition
   - Ad-hoc notes like rush_order flag
   - Useful for one-off data annotations

3. **Ellipsis Shorthand (…)**
   - Sparse data representation
   - Reduces file size when many trailing fields are empty
   - Improves readability

4. **Single-Line Delimiter Override**
   - Complex URLs with query parameters containing pipes
   - Data containing standard delimiter characters
   - Line-specific delimiter changes

5. **Nested Arrays with Secondary Delimiter (!)**
   - Arrays within fields using secondary delimiter
   - Permission sets, tag lists, hierarchical data
   - Default secondary delimiter is !

6. **Text Block References**
   - Multi-line content stored separately
   - Referenced from data fields
   - Keeps data clean and organized

## Usage

Parse this file with a SET file parser that supports v4.2 features.
For runtime calculations, your parser must implement the :: pattern.

Nested arrays require secondary delimiter support (section 1.3.3).
[EOG]

[EOF]

7.5 Environment-Specific Configuration


env_config.set

[THIS-FILE]
Version|4.2
Environment|production
[EOG]

[DATABASE_PRODUCTION]
Host|prod-db-01.example.com
Port|5432
Database|myapp_prod
User|prod_user
Password|[{DB_PROD_PASSWORD}]
PoolSize|50
Timeout|30
SSL|true
[EOG]

[DATABASE_STAGING]
Host|staging-db.example.com
Port|5432
Database|myapp_staging
User|staging_user
Password|[{DB_STAGING_PASSWORD}]
PoolSize|20
Timeout|30
SSL|true
[EOG]

[DATABASE_DEVELOPMENT]
Host|localhost
Port|5432
Database|myapp_dev
User|dev_user
Password|dev_password
PoolSize|5
Timeout|60
SSL|false
[EOG]

[CACHE_PRODUCTION]
Provider|redis
Host|prod-cache-01.example.com
Port|6379
TTL|3600
MaxMemory|2GB
[EOG]

[CACHE_STAGING]
Provider|redis
Host|staging-cache.example.com
Port|6379
TTL|1800
MaxMemory|1GB
[EOG]

[CACHE_DEVELOPMENT]
Provider|memory
TTL|300
MaxMemory|100MB
[EOG]

[{DB_PROD_PASSWORD}]
<encrypted_password_here>
[EOG]

[{DB_STAGING_PASSWORD}]
<encrypted_password_here>
[EOG]

[EOF]

7.6 Embedded SetTag Example

config.js with embedded SET configuration: 
javascript
// {SETTAG:AppConfig}
// [SETTINGS]
// AppName|My JavaScript App
// Version|1.0.0
// Debug|false
// {/SETTAG/}

const config = extractSetTag('config.js', 'AppConfig');
console.log(config.SETTINGS.AppName); // "My JavaScript App"
index.html with metadata: 
html
<!DOCTYPE html>
<html>
<head>
  <!-- {SETTAG:PageMeta}
  [META]
  Title|My Page
  Author|John Doe
  Keywords|demo,example,set-file
  Description|[{PAGE_DESC}]
  
  [{PAGE_DESC}]
  This is a demo page showing how SET files can be
  embedded in HTML documents using comment tags.
  {/SETTAG/} -->
  
  <title>My Page</title>
</head>
<body>
  <h1>Demo Page</h1>
</body>
</html>

---

8. Version History & Migration

Version 4.2 (December 2025) - Refinements & Clarifications

Changes from v4.0 to v4.2:

    • Enhanced Delimiter Documentation
- Clarified delimiter positions in Delimiters setting
  • Standard format: :[]:{}:|:\:…:!
  • Positions: preamble, group start, group end, field, text group start, text group end, ellipsis, secondary
    • Secondary Delimiter Explicit Support
- Added ! as default secondary delimiter for nested arrays
  • Documented in [THIS-FILE] Delimiters setting
  • Enables nested data structures within fields
    • Single-Line Delimiter Override Clarification
- Syntax: Line starts with preamble delimiter + single-use delimiter
  • Example: :!URL!https://example.com/api?param=value|other=value
  • Clearer documentation of use cases
    • Text Group Naming Consistency
- Standardized on [{TEXT-GROUP}] syntax throughout documentation
  • Consistent usage of curly braces in group names
  • Clearer distinction from regular groups
    • Documentation Improvements
- Better examples for all advanced features
  • Clearer explanation of implementation flexibility
  • Enhanced migration guidance
Migration from v4.0 to v4.2:

Files created with v4.0 are fully compatible with v4.2 parsers. No file changes required.

If using delimiters, update [THIS-FILE] to include secondary delimiter explicitly:

v4.0 format: 

[THIS-FILE]
Delimiters|:[]:{}:|:\:…
[EOG]
v4.2 format (recommended): 

[THIS-FILE]
Delimiters|:[]:{}:|:\:…:!
[EOG]

This makes the secondary delimiter explicit and enables better nested array support.

---

Version 4.0 (November 2025) - Major Simplification

Philosophy Change: Version 4.0 represented a fundamental shift toward simplicity and implementation flexibility. The format was simplified while maintaining backward compatibility with most v3.x files.

Major Changes:

    • Removed Mandatory Preamble
- v3.x: Required 4-7 line preamble with specific format
  • v4.0: Optional [THIS-FILE] group for configuration
  • Benefit: Simpler files, easier to get started
    • Eliminated Group Type Distinction
- v3.x: [=KEYVALUE=] syntax for key-value groups
  • v4.0: Just [GROUPNAME] - can contain positional or key-value data
  • Benefit: Less syntax to remember, cleaner files
    • Removed Comment Block Syntax
- v3.x: {|[COMMENT]|} ... {|[/COMMENT]|}
  • v4.0: Text outside groups is inherently a comment
  • Benefit: Simpler, more natural documentation
    • Added Features:
- Single-line delimiter override: :!field!field!field
  • Implicit EOG via empty lines (explicit [EOG] still allowed)
  • Clearer escape sequence rules (minimal: just \| and \\)
    • Simplified Escape Sequences
- v3.x: Required escaping [, ], {, }, space markers
  • v4.0: Only escape delimiter \| and backslash \\
  • Benefit: Less escaping needed, more readable
Migration from v3.x to v4.0:

Step 1: Update Preamble

v3.x format: 

filename.set
UTF-8
:[]:{}:|:\:…:
NFC|en-US|LTR


VERSION: 3.3
v4.0 format: 

filename.set

[THIS-FILE]
Version|4.0
Delimiters|:[]:{}:|:\:…:
Encode|UTF-8
Localize|NFC|en-US|LTR
[EOG]

Step 2: Update Group Names

v3.x format: 

[=SETTINGS=]
Key|Value
[EOG]
v4.0 format: 

[SETTINGS]
Key|Value
[EOG]

Simply remove the = signs from group names.

Step 3: Replace Comment Blocks

v3.x format: 

{|[NOTE]|}
This is a comment
{|[/NOTE]|}

[DATA]
v4.0 format: 

This is a comment

[DATA]
Or use unreferenced text blocks: 

[{NOTE}]
This is a comment
[EOG]

[DATA]

Step 4: Simplify Escape Sequences

v3.x: Required escaping brackets and braces 

Expression|\[value\] in \{range\}
v4.0: Only escape delimiter and backslash 

Expression|[value] in {range}

Unless the line starts with [, brackets don't need escaping.

Automated Migration Script:

python
def migrate_v3_to_v4(v3_filename, v4_filename):
    lines = read_file(v3_filename)
    output = []
    
    # Convert preamble to [THIS-FILE] group
    if is_v3_preamble(lines[0:7]):
        output.append(lines[0])  # filename
        output.append("")         # blank line
        output.append("[THIS-FILE]")
        output.append(f"Version|4.0")
        if lines[1].strip():
            output.append(f"Encode|{lines[1]}")
        if lines[2].strip():
            output.append(f"Delimiters|{lines[2]}")
        if lines[3].strip():
            output.append(f"Localize|{lines[3]}")
        output.append("[EOG]")
        output.append("")
        lines = lines[7:]  # Skip preamble
    
    # Convert group names
    for line in lines:
        # Remove [=NAME=] syntax
        line = re.sub(r'\[=(.+)=\]', r'[\1]', line)
        
        # Remove comment blocks
        if '{|[' in line and '|}'  in line:
            continue  # Skip comment block markers
            
        output.append(line)
    
    write_file(v4_filename, output)

Backward Compatibility:

v4.0+ parsers can read most v3.x files with these caveats:

  • Preamble must be converted to [THIS-FILE] group
  • Comment blocks are not supported (but can be converted to text outside groups)
  • [=NAME=] syntax works but is deprecated
v3.x parsers cannot reliably read v4.0+ files that use:
  • [THIS-FILE] group instead of preamble
  • Text outside groups as comments
  • Single-line delimiter override
---

Version 3.3 (November 2025)

  • Clarified progressive preamble definition
  • Standardized group naming rules
  • Updated [EOG] and [EOF] markers to optional
  • Enhanced documentation

Version 3.2 (November 2025)

  • Added key-value groups [=NAME=]
  • Added text block groups [{NAME}]
  • Added text block reference system
  • Enhanced validation rules

Version 3.0 (September 2025)

  • Added special functions (ellipses, single-use fields :::)
  • Enhanced internationalization support
  • Improved escape character handling
  • Added SetQL query language

Version 2.0

  • Core format specification
  • Escape sequences
  • Comment blocks
  • SetTag extensions
---

Migration Best Practices

When to migrate:

  • Creating new SET files → Use v4.2
  • Simple v3.x files → Easy to migrate to v4.x
  • Complex v3.x files with many comment blocks → Evaluate benefits
  • Production systems → Test thoroughly before migration
  • v4.0 to v4.2 → No migration needed, fully compatible
Testing migration:
    • Back up original files
    • Run migration script
    • Parse both versions with v4.2 parser
    • Compare data structures
    • Validate all references resolve
    • Test with your application
Gradual migration:
  • Migrate configuration files first (simplest)
  • Then data files
  • Finally, complex files with many text blocks
  • Keep old version files until new versions are validated
---

License

[![CC BY 4.0][cc-by-shield]][cc-by]

This guide is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

Copyright (c) 2025 Kirk Siqveland

You are free to:

  • Share — copy and redistribute the material in any medium or format
  • Adapt — remix, transform, and build upon the material for any purpose, even commercially
Under the following terms:
  • Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made
Full license text: https://creativecommons.org/licenses/by/4.0/

Implementations of this specification may use any license of the implementer's choosing.

[cc-by]: https://creativecommons.org/licenses/by/4.0/ [cc-by-shield]: https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg

---

End of SET File Implementation Guide v4.2*

Questions or feedback?
Visit: https://github.com/kirksiqveland/setfile

License:
Creative Commons Attribution 4.0 International (CC BY 4.0)
Copyright (c) 2025 Kirk Siqveland



Page last modified on January 31, 2026, at 07:58 PM