Methodology¶
TL;DR
Timeline0 investigates the Smartup Hypothesis using Engelbart's eight observable indicators of Collective IQ.
We specify in advance what evidence supports or refutes the hypothesis. We make our methods transparent. We design experiments others can replicate. We stay open to being wrong.
This section explains how.
🟢 Research Status: Established Foundation — Methodology built on Engelbart, Ostrom, STS theory, Cybernetics, and Connectivism.
The research question¶
The central question driving this research programme:
How can socio-technical systems augment the Collective IQ of communities working toward a shared mission?
Our methodology is designed to answer this question with evidence, not intuition.
Why rigor matters¶
A researcher can prove almost anything if allowed to choose what counts as evidence after the fact.
A researcher can explain away any failure if allowed to change the rules mid-experiment.
Scientific integrity requires specifying in advance:
- What counts as evidence for the hypothesis
- What counts as evidence against it
- How you measure it
- What threshold you're using
- How others can replicate the test
This section establishes those commitments before we examine the evidence.
The intellectual foundations¶
Timeline0 does not invent the framework for studying Collective IQ. It stands on five well‑established research traditions:
- Engelbart's augmentation philosophy, which gave us the eight observable indicators of higher and lower Collective IQ.
- Ostrom's commons governance, demonstrating that institutional design changes collective outcomes.
- Sociotechnical systems theory, proving that social and technical dimensions must co‑evolve.
- Cybernetics, showing how feedback loops and information flows shape system behaviour.
- Connectivism, explaining how knowledge emerges from connection‑making in transparent networks.
Each tradition has shaped the measurement approach; a full discussion of how they inform the hypothesis can be found in the Smartup Hypothesis chapter.
What we measure: Engelbart's Eight Indicators¶
Rather than using abstract proxies, we measure Collective IQ directly against the eight observable indicators Engelbart identified.
Each indicator can be observed, measured, and compared over time.
The Eight Indicators Operationalized¶
| Indicator | What We're Looking For | How We Measure It |
|---|---|---|
| 1. Learns more quickly | Speed of absorbing experience, acquiring knowledge, identifying patterns | Pre‑ vs. post‑iteration cycle time; rate of documented lessons captured; new skills demonstrated; error recurrence reduction |
| 2. Remembers its past better | Quality of institutional memory. Decisions, actions, and reasoning retrievable. | Wiki completeness score; session logs preserved and referenced; decision rationale traceability; onboarding time trend |
| 3. Integrates capabilities faster | Speed of bringing together diverse skills, perspectives, and innovations | Cross‑team collaboration rate; time from problem to integrated solution; new capability adoption latency |
| 4. Understands its own makeup better | Self‑awareness of structure, strengths, weaknesses, actual vs. intended function | Self‑assessment accuracy index; structure‑function alignment score; governance effectiveness reports; own‑system diagnosis quality |
| 5. Sees its environment more clearly | Perception of external reality with greater accuracy and nuance | Threat/opportunity detection latency; environmental scanning completeness; accuracy of external situation reports |
| 6. Recognizes threats and opportunities faster | Speed of detecting external change; depth of impact understanding | Time from external change → community awareness; time from awareness → response decision; response quality rating |
| 7. Generates plans more cleverly | Comprehensiveness and creativity of response plans; adaptation quality | Plan complexity and scenario coverage metrics; outcome‑vs‑plan variance; recovery time from failure; plan reusability score |
| 8. Coordinates resources more effectively | Less friction in deploying effort, time, money, attention; greater alignment | Task completion cycle time; resource allocation latency; rework percentage; team alignment survey results |
Supporting Evidence: What Success Looks Like¶
A brief summary of the expected improvements if the hypothesis is supported. For a detailed breakdown, see the full evidence table in the Hypothesis.
- Learning & Memory (1, 2): Iteration cycles accelerate, documentation becomes more useful, onboarding time drops.
- Capability Integration (3, 4): Cross‑functional collaboration increases, self‑assessments align with external observations.
- Environmental Awareness (5, 6): Threat detection latency falls, response quality improves.
- Planning & Coordination (7, 8): Task cycles shorten, rework decreases, resource allocation grows more efficient.
Falsifying Evidence: What Failure Looks Like¶
Similarly, a summary of what stagnation or decline would look like. Detailed falsification conditions are pre‑registered in the Hypothesis.
- Learning & Memory (1, 2): Iteration cycles do not accelerate, documentation completeness is high but Search‑to‑Creation Ratio does not improve, mistakes repeat despite being logged.
- Capability Integration (3, 4): Cross‑functional collaboration remains siloed, self‑understanding does not improve despite transparency.
- Environmental Awareness (5, 6): Threat detection latency does not improve, response quality remains poor.
- Planning & Coordination (7, 8): Task cycles do not accelerate (or slow), rework increases, team friction rises.
On reproducibility: Why it has to be public¶
For Timeline0 to be science (not just an interesting experiment), other communities must be able to conduct the same test independently.
This requires complete transparency across four dimensions:
Every governance protocol, decision process, permission structure must be published.
Why? So another community can ask: "If we follow these same rules, do we get the same results?"
The complete SmartupOS architecture (Element, Forgejo, Engelbot). All open-source and auditable.
Why? So another community can ask: "If we use this same apparatus, do we see similar patterns?"
How tasks are created. How work is assessed. How conflicts are resolved. Documented with rationale.
Why? So another community can ask: "If we follow this same workflow, do we improve against the eight indicators?"
Every decision, task, financial transaction. Public ledger, fully auditable. Raw data for external research.
Why? So external researchers can ask: "What patterns exist in the data? Do the indicators actually improve?"
On experimental controls: Phase gates protect validity¶
Smartup Zero progresses through four phases—Validation, Design, Production, and Organization—each with strict advancement requirements.
These gates are not bureaucratic delays. They are experimental controls that protect scientific validity.
graph LR
VALID["Validation<br/>Concept<br/>Proven"]
DESIGN["Design<br/>Blueprint<br/>Validated"]
PROD["Production<br/>MVP<br/>Tested"]
ORG["Organization<br/>Sustainable<br/>Model Proven"]
VALID -->|4 Gates<br/>Financial<br/>Organizational<br/>Documentation<br/>Democratic + Scientific| DESIGN
DESIGN -->|4 Gates| PROD
PROD -->|4 Gates| ORGWhy phase gates matter for science:
| Without Gates | With Gates |
|---|---|
| Variables confound. Changes come from hypothesis + scaling chaos + selection bias all at once. | Each phase generates clean evidence before scaling. Isolate variables. Test assumptions explicitly. |
| Can't distinguish signal from noise. | Signal clarifies. We measure what we actually changed. |
| Observations become anecdotal. | Observations become reproducible. |
On avoiding bias: Three defenses¶
Science has a problem: researchers unconsciously favor evidence supporting their hypothesis.
Timeline0 uses three mechanisms to defend against this:
Before interpreting evidence, specify:
- What counts as success
- What counts as failure
- What threshold we're using
- How we measure it
- What we expect to see
This methodology section is that pre-registration. It's public. It's dated.
We cannot change it retroactively without announcing the change.
Every major document is published with reasoning visible.
We explicitly invite challenge from:
- Other researchers
- Partner institutions
- Community members
- People who think we're wrong
Criticism makes us stronger, not weaker.
As evidence accumulates, we submit findings to:
- Academic peer review
- Community deliberation
- Regulatory scrutiny (if relevant)
- Other Smartup experiments attempting replication
We don't get to interpret our own evidence alone.
On learning: The continuous cycle¶
Timeline0 is not a one-time test. It is a continuous cycle of observation → interpretation → refinement → testing.
graph TD
OBS["Observation<br/>What actually<br/>happened?"]
INT["Interpretation<br/>What does it<br/>mean against<br/>the 8 indicators?"]
REF["Refinement<br/>What should<br/>we change?"]
TEST["Test<br/>Does the change<br/>improve the<br/>indicators?"]
PUB["Publication<br/>Share what<br/>we learned"]
OBS --> INT --> REF --> TEST
TEST --> OBS
PUB -.-> OBSAfter each iteration:
- We observe what actually happened
- We interpret the meaning carefully, with outside input
- We refine our understanding
- We test refinements
- We publish findings, not conclusions
The goal is not to prove the hypothesis.
The goal is to steadily improve our understanding of what actually helps communities become more collectively intelligent against Engelbart's eight indicators.
What this methodology protects¶
graph TB
HYP["The Smartup Hypothesis"]
RIGOR["Scientific Rigor<br/>Pre-registration,<br/>gates, external review"]
HONEST["Intellectual Honesty<br/>Welcome falsification,<br/>publish everything"]
REPRO["Reproducibility<br/>Others can run<br/>the same test"]
MEASURE["Measurement<br/>Against Engelbart's<br/>8 indicators<br/>not invented metrics"]
LEARN["Learning<br/>Evidence improves<br/>our understanding"]
RIGOR --> LEARN
HONEST --> LEARN
REPRO --> LEARN
MEASURE --> LEARN
HYP -.-> LEARNThis methodology is not designed to prove the Smartup Hypothesis.
It is designed to generate trustworthy evidence, whether that evidence supports or refutes the hypothesis.
Because trustworthy evidence is what advances science.
What is next?¶
We have now established:
- What we're testing — the Smartup Hypothesis
- What success looks like — improvement against Engelbart's 8 CIQ indicators
- What failure looks like — stagnation or decline on the indicators
- How we measure — operationalized observation of each indicator
- Why others can replicate — complete transparency on rules, code, workflows, data
- How we avoid bias — pre-registration, external review, phase gates
The next question is practical:
What does this test actually look like when it's running?
Ways to Participate¶
-
Join Smartup Zero
Become an participant in the first experiment.
-
Start Your Own Smartup
Take SmartupOS and run your own experiment.
-
Sponsor the Research
Support Timeline0 and Smartup Zero through funding.
-
Collaborate as Researcher
Work on the research programme itself.
References:
Engelbart, Douglas. Augmenting Human Intellect: A Conceptual Framework (1962)
Ostrom, Elinor. Governing the Commons: The Evolution of Institutions for Collective Action (1990)
Tavistock Institute. Sociotechnical Systems Theory
Beer, Stafford. Designing Freedom (1974)
Meadows, Donella. Thinking in Systems (2008)
Downes, Stephen. Connectivism and Connective Knowledge (2012)