Delays with extension reviews

Developers Developer Announcements
4

To: Andrew @corney, @thomthom, @eneroth3, @jiminybillybob, @Matheron

TL;DR

I’d like to propose two complementary tools that move first-pass validation out of your queue and into developers’ local workflows — before the Submit button is ever pressed. Both are designed to be community-built, open-source, and low-maintenance for your team.

Context: How Other Ecosystems Handle This

You’re not alone in this challenge. Every major extension/plugin marketplace has wrestled with the same surge-quality-security trilemma:

  • VS Code Marketplace (Microsoft): Runs automated malware scans on every submission and recently added secret scanning in their vsce CLI tool so developers catch credential leaks before upload. They also introduced a Trusted Publisher badge system to fast-track trusted vendors — directly analogous to your paid/subscriber prioritization.

  • Open VSX Registry (Eclipse Foundation): Just rolled out mandatory pre-publication security checks in early 2026 — scanning for namespace impersonation, accidental secrets, and known malicious patterns. Suspicious submissions are quarantined for manual review rather than blocked entirely. Their key insight: shift from reactive to proactive.

  • npm: The publish-please / npm audit ecosystem lets developers dry-run a publish and catch vulnerabilities, uncommitted changes, version mismatches, and sensitive files before anything reaches the registry. The prepublish hook enforces this automatically.

  • GitHub Marketplace Actions: Enforces automatic version-tag-to-manifest matching as a publish prerequisite. No mismatch, no publish. Simple, zero-review-team overhead.

The pattern is consistent: the most effective triage is the triage developers do themselves, when given good tooling and fast feedback loops.

Proposal 1 — ew-preflight: A Local Pre-Submit CLI

A command-line tool (Ruby gem) that developers run before packaging their .rbz. It wraps existing tools and adds EW-specific checks.

Tier 1 — Instant checks (< 5 seconds)

These run on every invocation and catch the most common rejection reasons:

Check What it catches
rubocop-sketchup --only SketchupRequirements Namespace pollution, exit!, $LOAD_PATH modification, Gem.install, missing SketchupExtension registration, SketchupRequire with file extension
RBZ structure validation Single root .rb + matching subfolder; root file only registers, doesn’t load
Version consistency check SketchupExtension.new version string vs. EW submission metadata — catches the 1.3.2/1.4.6 class of mismatch your team flagged
Metadata completeness Extension name uniqueness within local namespace; description non-empty; support URL present
Binary / compiled file detection Flags .so, .bundle, .dll files and prompts developer to confirm they have trusted-vendor status
Secret scanning Regex sweep for API keys, tokens, passwords in source (mirrors what VS Code’s vsce now does)

Tier 2 — Deep scan (opt-in, --deep flag, ~30–60 seconds)

Runs an LLM-assisted review against an EW-specific system prompt. Think of it as a local CodeRabbit for SketchUp extensions:

  • Checks for AI-generated “slop” patterns: generic boilerplate, lack of SketchUp API usage, copy-paste from tutorials
  • Flags use of undocumented SketchUp APIs
  • Detects obfuscated code blocks that would trigger your security concern about compiled/encrypted submissions
  • Reviews observer callbacks for missing start_operation with transparent flag
  • Can run locally via Ollama (no cloud dependency) or OpenAI-compatible endpoint

Output example

$ ew-preflight build/my_extension.rbz

EW Preflight v1.0 — AgiliCity
================================
✅  RBZ structure valid
✅  Single root .rb loader
✅  SketchupExtension registered (load-default: true)
✅  No global namespace pollution detected
⚠️  VERSION MISMATCH — loader: 2.1.0 / EW metadata: 2.0.9
❌  SketchupRequire: remove .rb extension on line 14 of main.rb
❌  Binary detected: libs/x64/my_solver.so — confirm trusted-vendor status
ℹ️  Run with --deep for AI-assisted quality review

2 errors, 1 warning. Fix errors before submitting.

Distribution

  • Ruby gem: gem install ew-preflight
  • GitHub Action: uses: sketchup-community/ew-preflight@v1 — runs on every push/tag
  • Works offline; no EW account required

Proposal 2 — Open a Subset of ew-review to the Community

We understand your internal ew-review tool is the gold standard, and we’re not asking for full access. A more modest ask:

Publish a read-only “developer preview” mode — a subset of your review checklist as a structured JSON/YAML spec that community tools (including ew-preflight) can consume.

Specifically:

  1. A machine-readable list of auto-rejection rules — the things that are always a hard NO, expressed as a schema developers can validate against locally. Version mismatch format, binary restrictions, namespace requirements, etc.

  2. A versioned “what changed” log — when rules change (like the current version-string policy), developers get notified via gem update rather than finding out after a 3-week queue wait.

  3. Optional: a sandboxed API endpoint — POST /ew-review/preflight that accepts an .rbz and returns the Tier 1 results your automated pipeline already produces, without entering the human-review queue. This is essentially what you described as “AI review on submission with feedback visible within a few minutes” — we’d just make it callable by developers before submission too.

This is the model the Open VSX Registry is moving toward, and it has the significant advantage that you control the spec — community tools stay in sync automatically when you update it.

What This Gives Your Team

Current situation With this tooling
Version mismatches discovered in review Blocked at preflight, developer fixes in minutes
Binary files from unknown developers enter queue Flagged locally, developer aware before submitting
AI slop detected manually Tier 2 scan surfaces low-effort submissions
New team member doesn’t know unwritten rules Rules are in the gem; always current
Human reviewer reads every SketchupRequire offense rubocop-sketchup already catches this; just needs a wrapper

The goal is not to replace your review — it’s to ensure that what reaches your queue has already passed the mechanical checks, leaving your reviewers free to focus on the things that actually require human judgment: originality, safety intent, UX quality.

Proposed Next Steps

  1. We build ew-preflight v0.1 (Tier 1 only) — open source, MIT license, community-maintained. We’re happy to build this without any commitment from your team.

  2. We’d welcome a 30-minute call to understand which of your current rejection reasons could be expressed as machine-readable rules. Even a rough list shared informally would let us cover 80% of the cases.

  3. If the sandboxed API endpoint (Proposal 2, item 3) is on your roadmap anyway, we’d love to be early testers.

We recognize you’re stretched thin right now, so we’ve deliberately designed this as something the developer community builds for you — not something that requires your team’s engineering time to maintain.

Thanks for all you do keeping the ecosystem healthy!

Disclaimer: I’m posting this on behalf of AgiliCity | Modelur, not as Extensioneers Guild member. But I’m pretty sure some Extensioneers would be interested in contributing if we are to move forward with this.