Registry of Registries

Introduction

The Registry Management and Access System is a comprehensive backend framework for managing, querying, and securely accessing distributed registry services. It provides an integrated solution that combines:

CRUD operations for dynamic registry creation, updates, and deletions
Query interfaces for advanced filtering and discovery via both REST and GraphQL
Health-aware socket proxying, enabling safe and real-time access to healthy registry nodes

This system is designed to power infrastructure platforms that require modular, decentralized registries representing services, agents, AI tools, or computational units. It ensures that all registry operations—from onboarding and metadata indexing to access routing—are governed by verified health status and reliable data models.

Key Objectives

Provide a MongoDB-backed registry layer with full metadata and health endpoint support
Offer flexible and powerful query interfaces to locate registries by ID, type, tags, or partial matches
Enable real-time access to registries via a socket-level proxy, guarded by health checks and caching
Support caching and performance optimization using Redis with TTL-based health status management

Who Is This For?

This system is intended for:

Developers managing a distributed ecosystem of services or agents
Platform engineers who require a safe and efficient proxy to forward traffic only to live registries
API developers who want unified access to create, search, and route to registry services
Infrastructure designers building a service-oriented architecture with runtime contract enforcement

Architecture

The Registries System is a modular service designed to manage, store, query, and monitor external or internal registries. It supports registry specification management, asset uploading, dynamic indexing, metadata search, health monitoring, and multi-modal query access (REST, GraphQL, and DSL-based). This section provides a detailed architectural breakdown of the system, categorized into key subsystems.

registry-of-registries

1. Registry Specification Lifecycle

This subsystem handles creation, update, and deletion of registry entries. The architecture is designed to support full validation, metadata persistence, and asset archival.

Components

Controller
Entry point for the create, update, and delete APIs.
Routes validated inputs to appropriate modules.
Spec Parser and Validator
Validates the structure and schema of incoming registry specifications.
Ensures conformance before storage or execution.
Create / Update / Delete Modules
Encapsulate business logic for respective operations.
Handle DB interaction and asset upload orchestration.
Assets Uploader
Uploads specification ZIP files to S3-compatible storage.
Works in tandem with the zip file reader to extract asset contents.
Assets DB Client API
Registers the uploaded specification as an asset within the asset registry.
DB Module
Updates the registry index and metadata in the internal database (DB index).

Flow Summary

Registry ZIP specification is submitted.
The spec is parsed and validated.
Assets are extracted and uploaded to object storage.
Metadata is stored, and the registry is registered in both the assets registry and DB index.

2. Index and Query Subsystem

This subsystem enables advanced querying over registered registries using indexed metadata. It supports generic REST queries, GraphQL queries, and DSL-based search execution.

Components

DB Index
Central database index storing searchable metadata from registry specifications.
DB Query System
Provides querying capabilities over the DB index.
Used by REST, GraphQL, and DSL query interfaces.
Generic Query REST API
Accepts structured query requests via standard REST endpoints.
Converts query to internal DB-compatible format for execution.
GraphQL API
Provides flexible querying using GraphQL syntax.
Powered by the GraphQL Middleware, which resolves query trees into DB queries.
DSL-Based Search API
Enables complex logic-based filtering and querying using a custom DSL.
Handled by the Search DSL Executor and Search DSL Engine, both of which translate logical expressions into database queries.

Flow Summary

Client sends a query via REST, GraphQL, or DSL.
The controller (or middleware) translates it to a DB-compatible query.
The DB Query System executes the query against the indexed metadata.

3. Health Check and Proxy Subsystem

This subsystem provides runtime health monitoring of registries and facilitates proxy access to internal or external registry endpoints.

Components

Health Checker
Actively probes the health of each registry using its health_check_api.
Health Cache with TTL (Redis)
Caches health results with a short TTL to reduce redundant checks.
Ensures responsiveness and load minimization.
Internal Health Checker
Uses the core database to locate registry health check APIs.
Triggers health checks and stores results in cache.
Internal Registry Proxy
Forwards data requests to registry endpoints.
Supports secure, internal-only communication as needed.
Registries Proxy
Acts as the outer gateway, combining caching, health checks, and proxy routing logic.

Flow Summary

A request arrives to access a registry.
The internal proxy checks the health cache (Redis).
If data is missing or expired, the health check is triggered.
Results are cached and passed to the client through the proxy.

Schema

The RegistryCoreData schema represents a single registry record in the system. Each registry holds metadata, routing configuration, documentation links, asset IDs, and a nested registry_search_data block for search indexing and classification.

This schema is persisted in the MongoDB registry_core_data collection.

RegistryCoreData – Field Reference

Field	Type	Description
`registry_id`	`str`	Unique identifier for the registry
`registry_url`	`str (optional)`	Public endpoint for accessing the registry
`registry_api_index`	`str (optional)`	Index or root path to the registry's API surface
`registry_documentation_s3_url`	`str (optional)`	S3 URL pointing to documentation zip
`registry_client_sdk_s3_url`	`str (optional)`	S3 URL pointing to downloadable client SDK
`registry_asset_id`	`str (optional)`	Reference to the asset record linked to this registry
`health_check_api`	`str (optional)`	URL to ping for checking if the registry is healthy
`registry_acl_config_info`	`dict[str, str]`	Access control metadata (e.g., visibility, auth mode, org ownership, etc.)
`registry_search_data`	`RegistrySearchData`	Embedded search indexing object used for querying

RegistrySearchData – Field Reference

Field	Type	Description
`registry_id`	`str`	Redundant reference to `registry_id`
`registry_type`	`str (optional)`	High-level category of the registry (e.g., "tool", "agent", "asset")
`registry_sub_type`	`str (optional)`	Sub-classification for internal grouping
`registry_search_tags`	`List[str]`	List of searchable tags associated with this registry
`registry_search_description`	`str (optional)`	Short searchable description used in UI and query filtering
`registry_metadata`	`dict[str, str]`	Additional structured metadata for search/lookup

Python Dataclass

from dataclasses import dataclass, field
from typing import Optional, List, Dict

@dataclass
class RegistrySearchData:
    registry_id: str
    registry_type: Optional[str] = None
    registry_sub_type: Optional[str] = None
    registry_metadata: Dict[str, str] = field(default_factory=dict)
    registry_search_tags: List[str] = field(default_factory=list)
    registry_search_description: Optional[str] = None

    def to_dict(self):
        return {
            "registry_id": self.registry_id,
            "registry_type": self.registry_type,
            "registry_sub_type": self.registry_sub_type,
            "registry_metadata": self.registry_metadata,
            "registry_search_tags": self.registry_search_tags,
            "registry_search_description": self.registry_search_description,
        }

    @classmethod
    def from_dict(cls, data):
        return cls(
            registry_id=data.get("registry_id", ""),
            registry_type=data.get("registry_type"),
            registry_sub_type=data.get("registry_sub_type"),
            registry_metadata=data.get("registry_metadata", {}),
            registry_search_tags=data.get("registry_search_tags", []),
            registry_search_description=data.get("registry_search_description")
        )


@dataclass
class RegistryCoreData:
    registry_id: str
    registry_url: Optional[str] = None
    registry_api_index: Optional[str] = None
    registry_documentation_s3_url: Optional[str] = None
    registry_client_sdk_s3_url: Optional[str] = None
    registry_acl_config_info: Dict[str, str] = field(default_factory=dict)
    registry_asset_id: Optional[str] = None
    health_check_api: Optional[str] = None
    registry_search_data: Optional[RegistrySearchData] = None

    def to_dict(self):
        return {
            "registry_id": self.registry_id,
            "registry_url": self.registry_url,
            "registry_api_index": self.registry_api_index,
            "registry_documentation_s3_url": self.registry_documentation_s3_url,
            "registry_client_sdk_s3_url": self.registry_client_sdk_s3_url,
            "registry_acl_config_info": self.registry_acl_config_info,
            "registry_asset_id": self.registry_asset_id,
            "health_check_api": self.health_check_api,
            "registry_search_data": self.registry_search_data.to_dict() if self.registry_search_data else None,
        }

    @classmethod
    def from_dict(cls, data):
        return cls(
            registry_id=data.get("registry_id", ""),
            registry_url=data.get("registry_url"),
            registry_api_index=data.get("registry_api_index"),
            registry_documentation_s3_url=data.get("registry_documentation_s3_url"),
            registry_client_sdk_s3_url=data.get("registry_client_sdk_s3_url"),
            registry_acl_config_info=data.get("registry_acl_config_info", {}),
            registry_asset_id=data.get("registry_asset_id"),
            health_check_api=data.get("health_check_api"),
            registry_search_data=RegistrySearchData.from_dict(data["registry_search_data"])
            if data.get("registry_search_data") else None
        )

You're welcome! Below is the ## Create, Update, Delete, and ZIP Upload APIs section with:

Endpoint descriptions
Sample payloads
cURL examples

Create, Update, Delete and ZIP Upload APIs

These endpoints manage the lifecycle of registries in the system. All operations persist data to the MongoDB registry_core_data collection. The ZIP upload API supports bundled asset ingestion and asynchronous background processing with Redis-based status tracking.

1. Create Registry

POST /registry/create Creates a new registry entry from a structured JSON spec.

Request Payload (`application/json`)

{
  "registry_id": "demo-registry",
  "registry_url": "https://demo.example.com",
  "registry_api_index": "/v1",
  "registry_documentation_s3_url": "s3://docs/demo.zip",
  "health_check_api": "https://demo.example.com/health",
  "registry_acl_config_info": {
    "visibility": "public"
  },
  "registry_search_data": {
    "registry_id": "demo-registry",
    "registry_type": "tool",
    "registry_search_tags": ["AI", "demo"],
    "registry_search_description": "Demo registry for testing"
  }
}

Sample cURL

curl -X POST http://localhost:8080/registry/create \
  -H "Content-Type: application/json" \
  -d @registry.json

2. Update Registry

PUT /registry/update Updates an existing registry by registry_id. Only specified fields will be overwritten.

Request Payload (`application/json`)

{
  "registry_id": "demo-registry",
  "registry_url": "https://demo-updated.example.com"
}

Sample cURL

curl -X PUT http://localhost:8080/registry/update \
  -H "Content-Type: application/json" \
  -d '{"registry_id": "demo-registry", "registry_url": "https://demo-updated.example.com"}'

3. Delete Registry

DELETE /registry/delete/<registry_id> Deletes the registry by ID.

Sample cURL

curl -X DELETE http://localhost:8080/registry/delete/demo-registry

4. ZIP Upload API (Async)

POST /registry/upload_zip Uploads a ZIP file containing assets and a registry spec, processes it in the background, and returns a task ID.

Multipart Form-Data Fields

Field	Type	Description
`spec`	JSON string	RegistryCoreData spec (as text)
`zip`	`.zip` file	ZIP file with docs/sdk/etc.

Sample cURL

curl -X POST http://localhost:8080/registry/upload_zip \
  -F 'spec={"registry_id":"demo-registry","registry_search_data":{"registry_id":"demo-registry"}}' \
  -F 'zip=@assets_bundle.zip'

5. Upload Status API

GET /registry/upload/status/<task_id> Checks the current status of a background ZIP upload.

Sample cURL

curl http://localhost:8080/registry/upload/status/<task_id>

Great — let’s now proceed with the next section:

Query APIs

The Query System allows clients to search registries using both structured REST endpoints and a flexible Mongo-style query interface. It supports exact matching, tag-based filtering, partial string matching, and advanced queries via raw MongoDB filters.

1. Get Registry by ID

GET /registry/query/by-id/<registry_id>

Retrieves the full registry document by ID.

Sample cURL

curl http://localhost:8090/registry/query/by-id/demo-registry

2. Get Registries by Type

GET /registry/query/by-type/<registry_type>

Returns all registries with a specific type (e.g., tool, agent).

Sample cURL

curl http://localhost:8090/registry/query/by-type/tool

3. Get Registries by Tags

POST /registry/query/by-tags

Filters registries where any tag in the provided list matches.

Request Payload

{
  "tags": ["AI", "compute"]
}

Sample cURL

curl -X POST http://localhost:8090/registry/query/by-tags \
  -H "Content-Type: application/json" \
  -d '{"tags": ["AI", "compute"]}'

4. Find by Partial URL Match

GET /registry/query/by-url?fragment=<text>

Performs a case-insensitive partial match on the registry_url.

Sample cURL

curl "http://localhost:8090/registry/query/by-url?fragment=openai"

5. Partial Match on Arbitrary Field

POST /registry/query/partial

Performs a partial (regex-based) match on any field.

Request Payload

{
  "field": "registry_search_data.registry_search_description",
  "fragment": "vision"
}

Sample cURL

curl -X POST http://localhost:8090/registry/query/partial \
  -H "Content-Type: application/json" \
  -d '{"field": "registry_search_data.registry_search_description", "fragment": "vision"}'

6. Generic Mongo Query

POST /registry/query

Performs a Mongo-style raw filter query on the collection.

Request Payload

{
  "filter": {
    "registry_search_data.registry_type": "tool",
    "registry_url": { "$regex": "openai", "$options": "i" }
  },
  "limit": 50
}

Sample cURL

curl -X POST http://localhost:8090/registry/query \
  -H "Content-Type: application/json" \
  -d @query.json

GraphQL API

The system exposes a GraphQL endpoint (/graphql) that allows clients to query registries using structured field selectors, filters, and search parameters. This provides a flexible alternative to REST with typed introspection and composability.

Endpoint

POST /graphql

The endpoint also supports the GraphiQL web interface for manual exploration.

Supported Query Operations

Operation	Parameters	Description
`registryById`	`registryId`	Fetch a single registry by ID
`registriesByType`	`registryType`	Fetch registries by type
`registriesByTags`	`tags: [String]`	Find registries containing any of the tags
`registriesByUrl`	`urlFragment`	Case-insensitive partial match on URL
`registriesByFilter`	`mongoFilter`, `limit`	Full Mongo-style query as a JSON string

Sample Queries

1. Get Registry by ID

query {
  registryById(registryId: "demo-registry") {
    registryId
    registryUrl
    registrySearchData {
      registryType
      registrySearchTags
    }
  }
}

2. Search by Tags

query {
  registriesByTags(tags: ["AI", "compute"]) {
    registryId
    registrySearchData {
      registrySearchTags
    }
  }
}

3. Search by Type

query {
  registriesByType(registryType: "tool") {
    registryId
    registrySearchData {
      registryType
    }
  }
}

4. Generic Filter

query {
  registriesByFilter(
    mongoFilter: "{ \"registry_search_data.registry_type\": \"tool\" }"
    limit: 10
  ) {
    registryId
    registryUrl
    registrySearchData {
      registrySearchDescription
    }
  }
}

GraphQL Schema (Simplified)

type RegistrySearchDataType {
  registryId: String
  registryType: String
  registrySubType: String
  registryMetadata: JSON
  registrySearchTags: [String]
  registrySearchDescription: String
}

type RegistryType {
  registryId: String
  registryUrl: String
  registryApiIndex: String
  registryDocumentationS3Url: String
  registryClientSdkS3Url: String
  registryAssetId: String
  healthCheckApi: String
  registryAclConfigInfo: JSON
  registrySearchData: RegistrySearchDataType
}

Proxy System

The Registry Proxy System is a socket-level proxy server that conditionally tunnels traffic to a target registry only if it passes a real-time health check. It is designed to ensure that clients never reach stale or unresponsive registries and enables dynamic routing based on registry health.

This proxy allows external clients to establish TCP connections and interact with registries transparently, while behind the scenes it performs:

Health cache lookup (via Redis)
Live health check (via HTTP/TCP probe)
MongoDB-backed registry endpoint resolution

Key Components

Component	Description
`SocketRegistryProxy`	Accepts and relays client TCP connections conditionally based on health
`HealthChecker`	Combines cached and live health status lookups
`HealthCache`	Uses Redis TTL cache to minimize repeated live checks
`InternalHealthChecker`	Performs live HTTP GET or TCP socket probe using metadata from MongoDB

Flow

Client connects to SocketRegistryProxy
First message must include the registry_id
Proxy checks Redis cache for recent health status
If no cache exists, it performs live check via InternalHealthChecker
If healthy, it opens a TCP socket to the target registry and relays data
If unhealthy, connection is dropped or error is returned

Environment Variables

Variable	Description
`REDIS_HOST`	Redis host (default: localhost)
`REDIS_PORT`	Redis port (default: 6379)
`MONGO_URI`	MongoDB connection string
`HEALTH_CACHE_TTL`	TTL in seconds for Redis health cache

Running the Proxy

python socket_registry_proxy.py

This launches the proxy on port 9000 by default and listens for client socket connections.

Sample Connection (Client)

import socket

s = socket.socket()
s.connect(("localhost", 9000))
s.send(b"demo-registry\n")  # registry_id as first message
s.send(b"GET /status\n")     # downstream traffic (e.g., HTTP request)
response = s.recv(4096)
print(response.decode())

Registry of Registries

Introduction

Key Objectives

Who Is This For?

Architecture

1. Registry Specification Lifecycle

Components

Flow Summary

2. Index and Query Subsystem

Components

Flow Summary

3. Health Check and Proxy Subsystem

Components

Flow Summary

Schema

RegistryCoreData – Field Reference

RegistrySearchData – Field Reference

Python Dataclass

Create, Update, Delete and ZIP Upload APIs

1. Create Registry

Request Payload (application/json)

Sample cURL

2. Update Registry

Request Payload (application/json)

Sample cURL

3. Delete Registry

Sample cURL

4. ZIP Upload API (Async)

Multipart Form-Data Fields

Sample cURL

5. Upload Status API

Sample cURL

Query APIs

1. Get Registry by ID

Sample cURL

2. Get Registries by Type

Sample cURL

3. Get Registries by Tags

Request Payload

Sample cURL

4. Find by Partial URL Match

Sample cURL

5. Partial Match on Arbitrary Field

Request Payload

Sample cURL

6. Generic Mongo Query

Request Payload

Sample cURL

GraphQL API

Endpoint

Supported Query Operations

Sample Queries

1. Get Registry by ID

2. Search by Tags

3. Search by Type

4. Generic Filter

GraphQL Schema (Simplified)

Proxy System

Key Components

Flow

Environment Variables

Running the Proxy

Sample Connection (Client)

Request Payload (`application/json`)

Request Payload (`application/json`)