NSF SBIR Phase I
Application Package

Recommended Title:

SBIR Phase I: Multi-Agent AI Architecture for Automated Creative Narrative Evaluation

Alternative Titles:

SBIR Phase I: Domain-Specific AI Evaluation of Long-Form Creative Text Using Coordinated Agent Ensembles

SBIR Phase I: Scalable Professional-Grade Evaluation of Creative Work Through Specialized Multi-Agent AI Systems

This Small Business Innovation Research Phase I project investigates novel methods for automated professional-grade evaluation of long-form creative narrative using coordinated multi-agent artificial intelligence architectures. The research addresses a fundamental challenge in natural language processing: developing AI systems capable of assessing the complex, multi-dimensional qualities of human creative work — structural integrity, narrative coherence, characterization depth, pacing calibration, voice consistency, and commercial viability — at the level of expert human evaluators.

The commercial need is clear and significant. Professional manuscript evaluation, screenplay coverage, and creative consultation services are structurally inaccessible to the 4.4 million independent authors, 400,000+ screenwriters, and 4.2 million podcasters operating outside institutional networks due to cost ($150–$1,500 per engagement) and availability constraints. Existing AI systems lack the specialized domain knowledge and multi-dimensional evaluation capability required to serve as reliable alternatives to expert human readers.

This project will investigate: (1) novel agent coordination architectures for parallel and sequential evaluation of long-form narrative; (2) methods for domain-specific calibration of scoring models to genre conventions and professional evaluation standards; (3) techniques for integrating stakeholder-specific evaluation criteria (MasterScore) into generalized AI evaluation pipelines without degrading core evaluation accuracy; and (4) validation methodologies for measuring the accuracy and reliability of AI creative evaluation against expert human benchmarks.

Successful outcomes will advance the scientific understanding of multi-agent AI evaluation systems while enabling commercial deployment of professional-grade creative evaluation tools accessible to independent creators — democratizing access to professional publishing and creative development infrastructure.

The Technical Innovation: Current AI systems evaluate text along simple dimensions (grammar, sentiment, readability). Evaluating creative narrative at a professional level requires coordinated assessment across structural, characterological, stylistic, market, and genre-specific dimensions simultaneously — a fundamentally different and significantly more complex task.

This research is innovative in three specific ways:

1. Multi-Agent Evaluation Architecture: Rather than a single model attempting to evaluate all dimensions of a creative work, this research investigates ensemble architectures in which specialized agents — each trained or prompted for a specific evaluation dimension — operate in coordinated parallel and sequential chains, with outputs synthesized into composite evaluations. The coordination protocols, handoff mechanisms, and synthesis methods are novel research contributions.

2. Domain-Specific Calibration for Creative Genres: Genre conventions in creative writing are complex, community-maintained, and not well-represented in general LLM training data. This research investigates methods for calibrating evaluation models to specific creative genres (literary fiction vs. genre romance vs. cozy mystery vs. science fiction) without separate fine-tuning for each genre — a significant generalization challenge.

3. Stakeholder-Defined Evaluation Integration (MasterScore): The ability to incorporate organization-specific evaluation criteria into a generalized evaluation pipeline without degrading core performance represents a novel challenge in adaptive AI evaluation systems. This research investigates integration architectures that maintain evaluation validity while accommodating heterogeneous stakeholder requirements.

Research Questions:

RQ1: What multi-agent coordination architectures maximize evaluation accuracy and consistency for long-form narrative text across structural, stylistic, and commercial evaluation dimensions?

RQ2: How can domain-specific creative genre calibration be achieved without per-genre fine-tuning of base evaluation models?

RQ3: What integration methods allow stakeholder-specific evaluation criteria to be incorporated into generalized AI evaluation pipelines with measurable accuracy and minimal performance degradation?

RQ4: What validation methodologies reliably measure the accuracy of AI creative evaluation against expert human evaluator benchmarks?

Technical Background and Prior Work:

Automated text evaluation has advanced significantly in recent years, with large language models demonstrating strong performance on standardized readability, coherence, and factual accuracy tasks. However, professional creative evaluation — the assessment of narrative craft, genre competence, characterization, pacing, and commercial viability in long-form creative work — represents a qualitatively different challenge that has received limited systematic research attention.

Existing automated manuscript feedback tools (ProWritingAid, Fictionary, AutoCrit) focus on surface-level features: grammar, word frequency, passive voice percentage, dialogue ratio. They do not address the deep craft dimensions that professional editors and literary agents evaluate: structural integrity, character arc completeness, narrative tension calibration, voice authenticity, and genre-specific convention adherence. ScriptBook applies ML to screenplay commercial viability prediction but operates as a black-box system without multi-dimensional craft evaluation.

The research gaps addressed by this project are: (1) coordinated multi-agent evaluation of complex creative text; (2) genre-specific calibration of evaluation models without per-genre fine-tuning; (3) integration of heterogeneous evaluation criteria into unified pipelines; and (4) validated benchmarking of AI creative evaluation against expert human readers.

Phase I Technical Approach:

Phase I research will proceed through four work packages:

WP1 — Agent Architecture Investigation (Months 1–2): Systematic evaluation of candidate multi-agent coordination architectures for creative text evaluation. Variables: sequential vs. parallel agent execution; centralized vs. distributed synthesis; agent specialization depth vs. breadth. Evaluation metric: correlation with expert human evaluator scores on a held-out benchmark dataset of professionally-evaluated manuscripts.

WP2 — Genre Calibration Methods (Months 2–4): Investigation of methods for genre-specific calibration without per-genre fine-tuning. Candidate approaches: retrieval-augmented genre context injection; few-shot genre exemplar prompting; genre-specific scoring rubric generation; dynamic calibration based on genre classification outputs. Evaluation metric: evaluation accuracy across 8 primary creative genres on held-out genre-labeled benchmark.

WP3 — Stakeholder Criteria Integration (Months 3–5): Investigation of architectures for incorporating organization-specific evaluation criteria (MasterScore) into the generalized evaluation pipeline. Research challenge: maintaining core evaluation accuracy while accommodating heterogeneous and potentially conflicting stakeholder criteria. Evaluation metric: accuracy preservation vs. stakeholder criteria incorporation trade-off curve.

WP4 — Validation and Benchmarking (Months 4–6): Development of validation methodology for AI creative evaluation. Construction of benchmark dataset through partnership with [2–3 literary organizations] whose professional readers evaluate a shared set of manuscripts. Correlation analysis between AI evaluation outputs and expert human evaluator scores. Inter-rater reliability analysis to establish expert human baseline for comparison.

Feasibility Demonstration: The AuthorSuite platform (authorsuite.online) represents a functioning proof-of-concept demonstrating that the core technical approach is viable. Phase I research will move from proof-of-concept to scientifically validated methodology — providing the research foundation for Phase II scaling and commercial deployment.

Market Opportunity:

The Total Addressable Market for professional creative evaluation services in the US is estimated at $4.8 billion annually, encompassing manuscript evaluation, script coverage, editorial consultation, and publishing strategy services currently consumed by individual creators and institutions. The independent creator segment — the primary target market — is growing at 17% annually driven by the expansion of self-publishing, streaming content, and podcast audiences.

Existing Commercial Traction:

AuthorSuite (authorsuite.online) is currently deployed in production with active subscribers, Stripe payment integration, and a functioning 31-agent evaluation pipeline. This demonstrates commercial feasibility and market demand prior to Phase I research — a significant advantage in the NSF SBIR evaluation process. Product lines: ASI Creative (consumer plans $27–$399/month or from $49 per evaluation), ASI Industry (publisher evaluation pools from $2,900/month and no-fee competition partnerships) and ASI Pro (professional reader plans from $399/month).

B2B Revenue Model — ASI Industry and ASI Pro:

ASI Industry licenses AuthorSuite's evaluation infrastructure to publishers and literary agencies through shared monthly evaluation pools (Imprint $2,900/month for 100 evaluations; House $5,000/month for 250), and partners with literary award programs, writing competitions and TV fellowship programs at no platform fee — by requiring certification for entry (entrants pay a discounted $25 per evaluation) or through a referral rebate. ASI Pro serves professional readers with monthly subscriptions from $399/month (10 evaluations) to $2,500/month (100 evaluations). This B2B model scales with partner submission volumes without requiring additional sales effort per evaluation — a highly efficient revenue model. Six market verticals are currently in active outreach: literary awards, TV fellowships, playwriting competitions, memoir contests, genre fiction organizations, and screenplay competitions.

Phase II Commercialization Path:

Phase I research outcomes directly enable Phase II commercial scaling: validated evaluation accuracy enables enterprise sales to literary agencies, publishers, and production companies; genre calibration methods enable vertical-specific product lines; MasterScore integration research enables institutional licensing at scale. Phase II request: $2,000,000 for 24 months of continued R&D and commercial deployment infrastructure.

Competitive Advantage:

Existing automated text tools (ProWritingAid, Fictionary, AutoCrit) are surface-level grammar and style checkers. No current competitor offers multi-dimensional, professional-grade creative narrative evaluation validated against expert human evaluators. Phase I research outcomes will establish a validated scientific foundation for AuthorSuite's evaluation claims — a significant competitive moat that commoditized AI writing tools cannot replicate.

This research will advance the scientific understanding of multi-agent AI evaluation systems with implications extending well beyond creative writing to any domain where expert human evaluation of complex, multi-dimensional human outputs is currently required but expensive or inaccessible: academic writing assessment, legal document review, clinical note quality evaluation, and product design feedback.

The direct societal impact centers on access and equity in creative professional development. The independent creator economy in America disproportionately excludes creators from non-coastal markets, working-class and middle-income backgrounds, and communities historically underrepresented in traditional publishing institutions. Professional editorial evaluation has been gated behind institutional affiliation and economic access — both of which correlate strongly with demographic privilege. Validated, accurate AI evaluation at accessible price points directly disrupts this access barrier.

The MasterScore system has a specific equity dimension: by enabling literary award programs and writing competitions to return structured feedback to all submitters — not just finalists — it addresses the feedback desert that non-advancing writers currently face. A writer who submits to 20 competitions and receives 20 form rejections has no basis for improvement. MasterScore makes meaningful feedback universally accessible.

The research will produce publicly available benchmark datasets and validation methodologies for AI creative evaluation — contributions to the broader NLP and AI evaluation research community. Publications resulting from Phase I work will be submitted to peer-reviewed venues in AI/ML and human-computer interaction.

Structure OS LLC, Ltd. is headquartered in Sheridan, Wyoming — a rural community without significant technology industry presence. This SBIR-funded research represents an investment in the rural technology sector, creating skilled employment in an underserved geographic market and demonstrating that high-value AI research and commercialization can succeed outside major tech hub geographies.

[PI Name], [Degree]
[Title], [Institution]
[Email] · [Phone]

Professional Preparation:
[Undergraduate Institution] · [Field] · [Degree] · [Year]
[Graduate Institution] · [Field] · [Degree] · [Year]
[Postdoctoral Institution] · [Field] · [Years] (if applicable)

Appointments:
[Current Position] · [Institution] · [Years]
[Prior positions as relevant]

Products (Publications):
[5 most relevant publications to this research — AI/ML, NLP, text evaluation, creative computing]

Synergistic Activities:
[Teaching, mentoring, broader impact activities — up to 5]

Collaborator and Co-Editor Information:
[Required disclosure per NSF format]

Don Swann II
Founder and Managing Director, Structure OS LLC, Ltd.
info@authorsuite.online · authorsuite.online
30 N Gould St Ste R, Sheridan, WY 82801

Role in Project: Business Lead / Commercialization Principal. Responsible for product development direction, B2B partner development, market deployment, and commercial operations. Serves as the primary point of contact between research activities and commercial application.

Relevant Experience:
[X years] in [creative industry / technology / entrepreneurship]. Founder of Structure OS LLC, Ltd. and developer of AuthorSuite — a live, production-deployed AI evaluation platform with active subscribers and a functioning 31-agent evaluation architecture. [Additional relevant experience as applicable: publishing industry experience, writing community engagement, technology product development, business development].

Entrepreneurial Track Record:
AuthorSuite: Live production platform · authorsuite.online · [launch date] · Active subscriber base · Stripe payment integration · 31-agent evaluation pipeline · B2B partner outreach across 6 market verticals

[Additional relevant products, ventures, or accomplishments as applicable]

SECTION A — SENIOR PERSONNEL

Principal Investigator (PI) — [Name, PhD]
[X]% FTE × $[annual salary] × 6 months = $[amount]
Responsible for research design, agent architecture investigation, methodology development, validation protocol, and publication preparation.

Don Swann II — Business/Commercialization Lead
[X]% FTE × $[annual compensation] × 6 months = $[amount]
Responsible for commercialization planning, B2B partner liaison, product integration, and reporting.

SECTION B — OTHER PERSONNEL

Research Assistant / Graduate Student — [Name or TBD]
[X]% FTE × $[stipend] × 6 months = $[amount]
Supports data collection, benchmark construction, literature review, and experimental implementation.

AI/ML Engineer — Contract
[Hours] × $[rate] = $[amount]
Implements technical research work packages, builds experimental agent architectures, develops evaluation pipeline modifications for research variants.

SECTION C — EQUIPMENT

GPU Computing Hardware (if not using cloud) — $[amount] or $0 if using cloud credits

SECTION D — TRAVEL

Conference presentations (NLP/AI venues — ACL, EMNLP, AAAI) — $[amount]
Industry partner meetings for benchmark data collection — $[amount]

SECTION E — PARTICIPANT SUPPORT

Expert reader participants for benchmark validation study — $[amount]
(Professional editors, literary agents, script readers compensated for benchmark evaluation participation)

SECTION F — OTHER DIRECT COSTS

Cloud computing (API inference costs, compute) — $[amount]
Software licenses — $[amount]
Materials and supplies — $[amount]
Publication costs (open access fees) — $[amount]

SECTION G — INDIRECT COSTS

[Rate]% of modified total direct costs = $[amount]

TOTAL PHASE I BUDGET: $275,000

Note: Exact line items require institutional negotiation with PI's institution and university indirect cost rate negotiation. This represents a representative allocation — exact figures to be determined with the PI institution.

Month 1 — Project Initiation and Literature Review
Milestones: SAM.gov and Research.gov registrations complete · PI and research team onboarded · Literature review of multi-agent AI evaluation systems completed · Benchmark dataset design finalized · Partner organization agreements for benchmark data collection signed

Month 2 — WP1: Agent Architecture Investigation (Phase 1)
Milestones: Baseline evaluation architecture documented · Three candidate coordination architectures implemented · Initial accuracy benchmarking completed against held-out evaluation set · Architecture selection criteria established

Month 3 — WP1 Completion + WP2: Genre Calibration (Phase 1)
Milestones: WP1 final architecture selected with documented rationale · Genre calibration candidate methods identified · Initial genre calibration experiments conducted on 4 primary genres · WP2 intermediate report prepared

Month 4 — WP2 Completion + WP3: MasterScore Integration (Phase 1)
Milestones: Genre calibration methodology finalized across 8 genres · MasterScore integration architecture candidates identified · Initial integration experiments conducted with 2 partner organizations · Q2 progress report submitted to NSF

Month 5 — WP3 Completion + WP4: Validation (Phase 1)
Milestones: MasterScore integration architecture finalized · Validation protocol finalized · Expert reader participants recruited and compensated · Initial validation study conducted · Correlation analysis between AI and human evaluator scores initiated

Month 6 — WP4 Completion + Final Reporting
Milestones: Validation study completed · Correlation analysis finalized · Phase I final report prepared · Phase II proposal drafted · Publication manuscript submitted to target venue · Benchmark dataset prepared for public release

Deliverables:
D1: Validated multi-agent evaluation architecture specification
D2: Genre calibration methodology documentation
D3: MasterScore integration protocol
D4: Benchmark dataset (with partner permission)
D5: Validation study results and correlation analysis
D6: Phase I final report for NSF
D7: Phase II proposal
D8: Peer-reviewed publication submission