CQ — features

A core tenet of CQ is to be a drop-in replacement for existing runtimes. The second is to enable users to run q/qsql against any dataset as easily as possible. To this end, CQ contains an internal table registry API where modules can attach global or namespaced tables from external systems. Point CQ at a bucket or metadata catalog and they'll show up as bona fide tables.

CQ is built in Rust with typed columns backed by Apache Arrow buffers for zero-copy Parquet/Arrow interop. It's designed to work directly with data in S3 buckets, either by scanning the directory structure or by connecting to an Iceberg metadata catalog.

Load Multiple Databases in One Session

Start CQ and run .cq.load[kind; ns; cfg]

param	type	description
kind	sym	kdb, parquet, or iceberg
ns	sym	desired namespace, or `. for global
cfg	dict or sym	config for the data kind

Example loading a Parquet database on S3 and a kdb database on local disk.

/ load s3 hdb to .aws namespace
.cq.load[`parquet;`.aws; `:s3://bucket/pqdb]

/ load local hdb to .local namespace
.cq.load[`kdb;`.local;`:/data/massive/us_stocks_sip/]

/ load iceberg catalog to .dw namespace
.cq.load[`iceberg;`.dw;([uri:"http://localhost:8181/catalog";warehouse:`massive])]

.cq.table[]
name                     kind    ns                source                                              columns partitions
-------------------------------------------------------------------------------------------------------------------------
.aws.day_aggs_v1         parquet .aws              s3://bucket/pqdb/day_aggs_v1/                       8       1548
.aws.minute_aggs_v1      parquet .aws              s3://bucket/pqdb/minute_aggs_v1/                    8       439
.dw.corp_act.dividends   iceberg .dw.corp_act      localhost:8181/catalog!corp_act.dividends           11      142
.dw.corp_act.ipos        iceberg .dw.corp_act      localhost:8181/catalog!corp_act.ipos                21      7
.dw.corp_act.splits      iceberg .dw.corp_act      localhost:8181/catalog!corp_act.splits              6       127
.dw.economy.inflation    iceberg .dw.economy       localhost:8181/catalog!economy.inflation            7       5
.dw.economy.labor_market iceberg .dw.economy       localhost:8181/catalog!economy.labor_market         5       5
.dw.reference.holidays   iceberg .dw.reference     localhost:8181/catalog!reference.holidays           6       13
.dw.reference.tickers    iceberg .dw.reference     localhost:8181/catalog!reference.tickers            18      187
.dw.day_aggs_v1          iceberg .dw.us_stocks_sip localhost:8181/catalog!us_stocks_sip.day_aggs_v1    9       128
.dw.minute_aggs_v1       iceberg .dw.us_stocks_sip localhost:8181/catalog!us_stocks_sip.minute_aggs_v1 9       128
.local.day_aggs_v1       kdb     .local            /data/massive/us_stocks_sip/day_aggs_v1             8       1598
.local.minute_aggs_v1    kdb     .local            /data/massive/us_stocks_sip/minute_aggs_v1          8       1597
..

Connecting to AWS services requires setting up AWS environment variables.

# setup aws credentials in bash env (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
eval "$(aws configure export-credentials --format env)"
export AWS_REGION=$(aws configure get region)

Join Across Multiple Databases

Example getting top 10 securities by volume notional from .aws.day_aggs_v1 (parquet on s3) and joining company name by ticker from .dw.reference.tickers (iceberg).

(10#`vol_ntnl xdesc select ticker, vol_ntnl:volume*close from .aws.day_aggs_v1 where date=2026.04.01)
       lj 1!select ticker, name from .dw.reference.tickers where date=2026.04.01
ticker vol_ntnl     name
-------------------------------------------------------------
SPY    6.410965e+10 State Street SPDR S&P 500 ETF Trust
QQQ    4.641474e+10 Invesco QQQ Trust, Series 1
NVDA   2.95492e+10  Nvidia Corp
MU     2.7379e+10   Micron Technology, Inc.
TSLA   2.237377e+10 Tesla, Inc. Common Stock
SNDK   1.546363e+10 Sandisk Corporation Common Stock
META   1.367454e+10 Meta Platforms, Inc. Class A Common Stock
IWM    1.28014e+10  iShares Russell 2000 ETF
GOOGL  1.120698e+10 Alphabet Inc. Class A Common Stock
MSFT   1.086583e+10 Microsoft Corp

Query Relational Databases with SQL

CQ is equipped with ADBC for column-by-column data transfer in Apache Arrow format, eliminating the cost of serializing and transforming row-by-row data into vector form, as required with ODBC.

conn:.sql.open[`driver`uri!(`adbc_driver_postgresql;"postgresql://me:secret@localhost/mydb")]
.sql.query[conn; "select id, sym, px, ts from quotes order by id"]
id sym  px    ts
-- ---- ----- ----------
1  AAPL 191.2 2024.01.02
2  MSFT 412.5 2024.01.02
3  GOOG 175.4 2024.01.03

/ sub function for auth and subscribe handshake
.u.sub:{[h]
    h .j.j ([action:"auth";params:apikey]);
    h .j.j ([action:"subscribe";params:"A.*"]);
    :1b
};

res:();

/ upd function to decode json payload
.u.upd:{[p]res,:.j.k p};

/ load from websockets
h:.cq.websocket[([uri:"wss://delayed.massive.com/stocks"; sub:`.u.sub; upd:`.u.upd])]

/ close session
hclose h

Developer Experience Improvements

Features that make development easier.

exec ... from on a parted table
} can end on a new line without a leading space
Copy/paste multi-line code into REPL
Built-in REPL history, no rlwrap required

Load Multiple Databases in One Session

Join Across Multiple Databases

Query Relational Databases with SQL

Subscribe to Real-Time WebSocket Streams

Developer Experience Improvements