Drolta: A DSL for SQLite in Social Sim Games

Technologies	Python, ANTLR4, PyTest
Project Status
My Role	Sole Developer

Drolta is an experimental SQLite query engine for wiring more simplified, composable, and declarative queries using a logic programming-inspired syntax. It was initially developed to simplify querying social simulation data generated by NPC interactions in simulation-focused narrative games.

Drolta was my third notable attempt at building a toolkit to help with searching through large populations of simulated characters and event data from my social sims like Neighborly and Minerva. The first was a Electron/ReactJS-based visual scripting tool for writing DataScript queries. The second was a simulation-specific Python solution that used data frame joins in Pandas. Each project got me closer to building Drolta, giving me experience in constraint satisfaction and transpiler/parser design.

This page is a quick blurb about my thoughts during and after development. If you’re looking for documentation on how to use Anansi, please refer to the README in the GitHub repository. Drolta is essentially on hiatus right now. I plan to resume development if I have a project that could actively use it. There are still some lingering features and bugs, most notably the need to add support for recursive rule definitions.

Design Problem and Motivation

This project explored the question: “How do you design a query language that is more intuitive than SQL, and allows users to query complex, graph-like data without requiring SQL experience?”

Drolta was designed to help game designers work with character data within simulation-driven emergent narrative games. So, simulation games like Crusader Kings, Civilization, World Box, and Dwarf Fortress. Drolta was developed to help with my dissertation research on story sifting. It helps you look for arbitrarily complex patterns in characters’ social connections or event histories.

I wanted an alternative to raw SQL that could be accessible to a broader audience of game designers. Drolta draws inspiration from logic programming languages and technologies such as Prolog, Datomic, and TED. It abstracts away the complexities of joining tables, allowing users to focus on specifying which variables they care about and how they should be matched (unioned) in the query.

My past query engine projects either could not work with real-time data or were not performant enough for real-time use. Drolta gave me the best of both worlds. It was performant enough for real-time use while remaining system-independent, enabling offline data exploration.

The Development Process

Drolta started as MQL (Minerva Query Language). I was working on my Minerva dynasty simulation at the time. Minerva used SQLite as a queryable backend for data. This was my way of facilitating both data analysis and queryability. The first version of Drolta worked much like visual programming languages like Blockly. Rule calls were facades for string templates. There was some simple parsing, and data was dropped into a template and passed up the template call stack. So a single MQL query would be transpiled one-to-one to an analogous SQL query.

MQL became drolta when I was encouraged by one of my PhD Advisors, Michael Mateas, to expand the language to enable people to author new rules directly in it. The old version required users to write template-resolution code that would accept a set of parameters and return a valid SQL string. At that point, I began developing Drolta as a full query language and not just a string generator.

I started development using a Google Doc to outline syntax and features. Then I used this document to write unit tests and the final documentation. It took me about three weeks to get a working prototype of the language. In my opinion, this was one of my best executed projects.

How it Works

This is a brief overview of how Drolta works. As always, the best way to understand it is by reviewing the source code. Drolta has two parts: the parser and the interpreter. The parser was generated using an ANTLR4 grammar and customized to produce a Drolta abstract syntax tree. The AST is processed by the interpreter to update the query engine state or evaluate a query. The engine state stores rule definitions and table aliases for use during query evaluation.

Queries and rules have roughly three sections: the header, the body, and the post-processors list. The header specifies what variables the query/rule needs to bind and return. Each matching combination is stored as a row in the table returned by the query/rule. The columns in the result table correspond to the variables in the header. The body contains statements that retrieve and filter data from rules or database tables. Finally, the post-processors specify how data should be organized, grouped, or limited (same as SQLite).

Drolta is only designed for querying information. It does not support creating or writing tables (except for temporary tables constructed during query evaluation). Nevertheless, Drolta is data-agnostic and can be used for any SQLite-based project, not just games.

Lessons Learned

1. Starting with documentation helps with test-driven development

Before writing any code for Drolta, I started outlining the language features in a Google Doc. I shared this Google Doc during meetings with my advisors, and it became somewhat of a shared digital chalkboard where we tested how language features should look and feel. This Google Doc eventually became the documentation found in the repository. The greatest benefit of doing all the planning upfront is that the documentation helped me write unit tests. Drolta is one of my best-executed open-source projects.

2. Think broader than games

MQL was intended to be a project-specific solution. However, during the transition into Drolta, I realized that this project could be much more general-purpose. This led me to frame Drolta more as a generic solution for complex queries rather than solely for games or a particular game. While Drolta hasn’t seen any adoption or traction online. I’m very proud of the package I created, and I look forward to potentially using it in the future.

3. Parse trees and abstract syntax trees are not the same thing

This lesson is personal to me. Using ANTLR as my parser generator, I was confused about how to connect my interpreter to the parser’s output. Some examples online skipped the AST abstraction and used the parse tree directly in the program. This led me to believe that the parser outputs an AST. After a week or so, I realized that ANTLR actually returns a parse tree, and that it is the developer’s responsibility to convert it into an abstract syntax tree.