Phase 1 · Foundation/Silo Engine is in active architecture & implementation

Sharpnr · Self-hosted data infrastructure

Infrastructure,
not business logic.

Silo Engine is Sharpnr's self-hosted platform for object storage, PostgreSQL provisioning, and data lake infrastructure. Organizations create the resources they need without ever touching the servers, RAID layout, or storage paths underneath them.

Rust + AxumPostgreSQLRAID-backedOpenAPI
~1 PB
RAID-backed storage
160 GB
Combined memory
10
Combined CPUs
1
Rust binary, systemd

Self-hosted physical infrastructure. systemd and Nginx. Kubernetes deliberately out of scope for the first version.

The boundary

Sharpnr Engine owns business logic.
Silo owns infrastructure and data.

Silo is not the Hospitality, Accounting, or HR database. It is the layer those applications provision resources from — and the line between the two is the single most important decision in the architecture.

Silo Engine owns

  • File storage & metadata
  • Storage paths & backends
  • Storage quotas
  • Checksums & compression
  • PostgreSQL provisioning
  • Database credentials
  • Database explorer APIs
  • Data lakes & warehouses
  • Data ingestion
  • Backups & snapshots
  • Audit logging
  • Resource access control

Sharpnr Engine owns

  • Hospitality
  • Accounting
  • HR
  • CRM
  • Other business modules

Business applications consume Silo resources. They do not reimplement them.

Platform

Three resources.
One foundation.

Storage, database, and data services are built on the same abstraction rather than as unrelated systems bolted together.

Files, objects, and CDN delivery on storage Silo owns.

File & object storage
CDN-backed delivery
Streaming compression (Zstd)
Checksums & integrity
Org / employee quotas
Backups & snapshots

Object storage

A URL that outlives
the filesystem.

Public addresses are deliberately independent of where the bytes physically live. Move to a different RAID array, a different backend, or a different datacentre — the URL does not change.

Public object

https://cdn.sharpnr.com/org_123/employee_456/profile/avatar.png

Quotas are enforced in Silo, not in the caller

Organization100 GB
Employee A20 GB
Employee B10 GB
Request pathpublic read
Public URL
CDN / Nginx
Silo Engine
Storage abstraction
Physical storage
Storage backendtrait

StorageBackend

├── LocalRaidStorage // initial target

├── S3Storage

├── CephStorage

└── …

Checksums

Computed on the upload stream for integrity, dedupe, and audit.

Compression

Zstd first, behind a trait. Already-compressed formats are skipped.

PostgreSQL as a service

A database, not
a superuser.

Organizations receive scoped roles and a connection string. The physical cluster stays an implementation detail — and nobody gets superuser.

ProvisioningPOST /v1/databases

├── Create database

├── Create PostgreSQL roles

├── Apply permissions

├── Generate credentials

└── Return connection information

Returned

postgres://org_123_app:••••••••@silo-engine-db.sharpnr.com:5432/org_123_db

?sslmode=require

Scoped rolesleast privilege
org_123_appruntime reads & writes
org_123_migratorschema changes
org_123_readonlyreporting
org_123_adminorg-level administration
postgresnever issued

Sharpnr employees are application identities. Silo does not mint a PostgreSQL login for every employee that happens to exist.

Database explorerv1 · read-oriented
prod-postgres
└─schemas
└─public
└─tables
└─orders
└─columns · 14
└─constraints · 3
└─indexes · 4
└─rows
└─customers
└─views
└─functions

No arbitrary SQL in v1.

The explorer exposes controlled operations over schemas, tables, views, functions, indexes, and constraints. A full SQL workspace comes later, once the controlled explorer is mature.

Browser → HTTPS → Silo Engine → PostgreSQL

Browser ⇸ PostgreSQL — never direct

Audit logappend-oriented
file.uploadedfile.downloadedfile.deleteddatabase.createddatabase.permission_changeddatabase.schema.modifiedstorage.quota_changedlake.createddataset.ingested

Lakes & warehouses

Beyond a
drive clone.

Raw data lands in a lake, gets transformed, and becomes an analytical warehouse — all on the storage foundation Silo already manages.

01

Sources

PostgreSQL · files · events · jobs

02

Ingest

Scheduled & on-demand

03

Data Lake

raw / processed / curated

04

Transform

ETL / ELT to Parquet

05

Warehouse

Query & analytics

Lake layoutacme-analytics

raw/

orders/ · customers/ · events/ · payments/

processed/

orders/ · customers/

curated/

daily_sales/ · customer_metrics/

lake://acme/orders/year=2026/month=08/day=10/

Partitioning, retention, and lifecycle are Silo's job — not yours.

Dataset formatsparquet primary
ParquetJSONJSONLCSVAvroORC

Parquet is columnar and pairs well with analytical query engines, so it leads for analytical workloads.

Reuse, don't reinvent.

Silo leans on mature open-source engines rather than building an analytical database from scratch. DuckDB is the early candidate for querying Parquet directly; a distributed engine only arrives if the workload genuinely demands one.

Engineering

How it is
actually built.

A single Rust crate, an explicit dependency graph, and an OpenAPI contract at the edge.

Rust · Axum · Tokio

A single asynchronous service. Async tasks over the Tokio runtime — never an OS thread per request.

Storage behind a trait

LocalRaidStorage today; S3, Ceph, or anything else later. The public API does not change when the disks do.

Streaming everywhere

Reader → compressor → writer. Multi-gigabyte objects are never loaded whole into memory.

Explicit dependency injection

Config, pools, and backends are constructed once at startup and passed in. No DI framework, no service building its own dependencies.

One EngineConfig

Environment read once inside the config module, validated at startup, injected everywhere else.

OpenAPI as the contract

Swagger UI and ReDoc ship with the service. The spec is the interface between Silo and every consumer.

Append-oriented audit log

Durable infrastructure events with organization, actor, resource, and timestamp. Sensitive payloads stay out.

UUID v7 identifiers

Time-ordered by default, so identifiers index well without a separate sort key.

Design principles

  • Infrastructure is separated from business logic.
  • Storage sits behind a backend interface.
  • Configuration is loaded once and injected.
  • Public URLs never depend on physical filesystem layout.
  • Large files are processed as streams.
  • Distributed infrastructure arrives only when it is needed.
  • Existing open source is reused before anything is reinvented.
Documentationopenapi contract
/swagger-uiInteractive explorer
/redocReference documentation
/v1/openapi.jsonMachine-readable spec

The OpenAPI document is the contract between Silo and every consuming Sharpnr service.

Roadmap

Incremental,
on purpose.

Infrastructure is added only when the storage and operational model underneath it can carry the weight. Not every phase is committed to the first release.

01FoundationConfigurationAxum serverApp state / DIError handlingLogging & tracingIn progress
02StorageStorage backendObject metadataUpload / download / deleteChecksumsCompressionQuotasNext
03AccessAuthenticationAuthorizationCDN integrationAudit loggingPlanned
04PostgreSQLProvisioningRoles & credentialsPublic DB endpointDatabase explorerBackups & snapshotsPlanned
05Data LakeLake creationDataset managementParquetIngestionRetentionPlanned
06Data WarehouseWarehouse creationLake → warehouseETL / ELTQuery engineAnalytics APIsPlanned
07Advanced Data ServicesSQL workspaceSearchVector storageAI / RAG infrastructurePlanned

Comparison

Where Silo fits —
and where it doesn't.

Silo is not trying to out-scale a hyperscaler. It is trying to make the infrastructure an organization already owns behave like a product. Phases mark what is built versus committed.

Comparison of Silo Engine against hyperscaler, backend platform, and self-assembled approaches to storage and database infrastructure
CapabilitySilo EngineSelf-hosted, one Rust serviceHyperscalere.g. S3 + managed PostgresBackend platforme.g. Supabase-styleSelf-assemblede.g. MinIO + Postgres + glue
Where the bytes liveYour hardwareVendor regionVendor or self-hostYour hardware
Object storage + CDN deliveryPhase 2–3MatureIncludedAssemble yourself
PostgreSQL provisioned via APIPhase 4MatureIncludedManual
Controlled explorer, no raw SQLPhase 4, by designNot the modelRaw SQL editorBring your own client
Org + per-employee storage quotasPhase 2, first-classBucket policy, not per-personNot modelledBuild it
Data lake + warehouse on same storagePhase 5–6Separate productsOut of scopeBuild it
One audit trail across storage + DBPhase 3Split per servicePartialBuild it
Egress billingNone — your networkMeteredMetered when hostedNone
Kubernetes requiredNo — systemd + NginxN/A, vendor-runNoOften, at scale
Who carries the operational loadYou run one binaryVendorVendor or youYou run every part

Phase labels refer to the Silo roadmap. Other columns describe categories of approach, not specific vendor SKUs.

Choose Silo when

  • You already own the hardware and want the storage bill to stop being a line item.
  • Quotas need to exist per organization and per employee, not per bucket.
  • One audit trail should cover file access and database changes together.
  • Business applications should provision infrastructure, not reimplement it.

Choose a hyperscaler when

  • You need multi-region replication and elastic burst capacity today.
  • There is no hardware and no team to operate it.
  • Compliance requires certifications a self-hosted deployment cannot inherit.

Be honest: not yet Silo when

  • You need a production data lake or warehouse this quarter — those are Phase 5 and 6.
  • You want an arbitrary SQL workspace on day one; v1 is deliberately a controlled explorer.
  • You only need one piece, and a single component you already run would do.

Questions

Frequently
asked.

Answers are drawn from the Silo Engine architecture document. If something here is out of date, the document is the source of truth.

Machine-readablefor AI crawlers
/llms.txtConcise index
/llms-full.txtFull architecture

What is Silo Engine?

Silo Engine is Sharpnr's self-hosted infrastructure platform. It provides file and object storage with CDN-backed delivery, PostgreSQL provisioning, a controlled database explorer, and — on the roadmap — data lakes, data warehouses, and analytics infrastructure. It is the layer Sharpnr Engine and other Sharpnr applications provision resources from.

Is Silo Engine available to use today?

Not yet. Silo Engine is in Phase 1, Foundation — the initial architecture and implementation phase. Configuration, the Axum server, application state and dependency injection, error handling, and tracing are the current focus. Storage lands in Phase 2 and PostgreSQL provisioning in Phase 4. Contact Sharpnr for early access.

How is Silo Engine different from Sharpnr Engine?

Sharpnr Engine owns business logic — Hospitality, Accounting, HR, CRM, and other business modules. Silo Engine owns infrastructure and data resources: storage, storage backends, quotas, checksums, compression, PostgreSQL provisioning, credentials, data lakes, backups, and audit logging. Silo is deliberately not the database for any business domain.

Which databases can Silo provision?

PostgreSQL. Silo creates the database, creates organization-scoped PostgreSQL roles, applies permissions, generates credentials, and returns connection information. The physical PostgreSQL server or cluster is an implementation detail.

Do organizations get PostgreSQL superuser access?

No. An organization never receives superuser credentials. Silo creates organization-scoped roles such as org_123_app, org_123_migrator, org_123_readonly, and org_123_admin. Sharpnr employees are application-level identities, so Silo does not create a PostgreSQL login for every employee.

Can I run arbitrary SQL in the Silo database explorer?

Not in v1. The explorer deliberately exposes controlled operations over schemas, tables, views, functions, indexes, constraints, and rows. An SQL workspace is planned for Phase 7, after the controlled explorer is mature. The browser never connects directly to PostgreSQL — it talks to Silo Engine over HTTPS, and Silo talks to PostgreSQL.

Does Silo Engine require Kubernetes or containers?

No. The initial deployment target is self-hosted physical infrastructure running as a normal service under systemd, with Nginx in front. Container orchestration is intentionally out of scope for the first version, though the architecture avoids assumptions that would block future horizontal scaling.

Where does Silo store data?

On RAID-backed storage on Sharpnr infrastructure — roughly 1 PB across multiple machines on a local network switch, with about 160 GB of combined RAM and 10 combined CPUs. Storage sits behind a StorageBackend abstraction, so S3, Ceph, or other implementations can be added without changing the public Silo API.

Are public file URLs tied to the filesystem layout?

No, deliberately. A public object is addressed as https://cdn.sharpnr.com/{org_id}/{employee_id}/{folder}/{filename}, which is independent of the physical storage path. The request travels from the public URL through CDN/Nginx to Silo Engine, then through the storage abstraction to physical storage.

What data formats does the Silo data lake support?

Parquet is the primary format under consideration because it is columnar and works well with analytical query engines. JSON, JSONL, CSV, Avro, and ORC are also planned. For early analytical and query workloads, DuckDB is a candidate for querying Parquet directly.

How does Silo handle very large files?

By streaming. Compression is designed as reader to compressor to writer so multi-gigabyte objects are never loaded whole into memory, with Zstandard as the initial algorithm behind a trait-based abstraction. Checksums are computed on the upload stream for integrity verification, duplicate detection, and auditing. Long-running work moves to background workers rather than blocking the request.

What is Silo Engine built with?

Rust, Axum, and Tokio, with PostgreSQL for metadata and OpenAPI as the published contract. Documentation is served at /swagger-ui, /redoc, and /v1/openapi.json. Configuration is centralized in a single EngineConfig loaded once at startup, and dependencies are injected explicitly rather than through a DI framework.

Infrastructure your organization creates, manages, and consumes.

Without ever managing the physical infrastructure underneath it. Silo Engine is early — talk to us about what you need from the storage and database layer.

Self-hosted · Rust · PostgreSQL · No Kubernetes required