Metadata-Version: 2.3
Name: analytio
Version: 0.1.0
Summary: CLI tool for profiling CSV datasets and evaluating data quality
Author: Adrián
Author-email: Adrián <adribarrios3205@gmail.com>
Requires-Dist: charset-normalizer>=3.4.9
Requires-Dist: fastexcel>=0.20.2
Requires-Dist: pandas>=3.0.5
Requires-Dist: polars>=1.43.1
Requires-Dist: fastapi>=0.141.1 ; extra == 'api'
Requires-Dist: pydantic>=2.13.4 ; extra == 'api'
Requires-Dist: python-multipart>=0.0.18 ; extra == 'api'
Requires-Dist: uvicorn[standard]>=0.12.0 ; extra == 'api'
Requires-Dist: pyside6>=6.11.1 ; extra == 'desktop'
Requires-Python: >=3.13
Provides-Extra: api
Provides-Extra: desktop
Description-Content-Type: text/markdown

# Analytio

Analytio helps you explore CSV files without writing code. Inspect column
types, missing values, statistics, value frequencies and charts before deciding
how to use your data.

The project includes a React web interface, a FastAPI HTTP API, a PySide6 desktop
application and a command-line tool. They reuse the Python analysis engine.

## License and source availability

Analytio is proprietary software. The current beta application is free to use
for lawful personal or professional purposes under the [LICENSE](LICENSE).
The original source code is private; application use does not grant permission
to publish, modify, reuse or redistribute that code or distribute modified builds.
Third-party components retain their respective licenses and rights.

A future open-source release would require a separate decision. Public access
to the beta application does not mean public access to its source repository.
Client-side JavaScript is necessarily delivered to the browser; keeping the
repository private is not a guarantee that distributed application code cannot
be inspected.

## Developer documentation

Start with [CURRENT.md](CURRENT.md) for the dated development status and beta gates,
[ARCHITECTURE.md](ARCHITECTURE.md) for the system map, and [AGENTS.md](AGENTS.md)
for repository instructions. Detailed operational guides live with the
[frontend](frontend/README.md) and [Windows packaging](packaging/README.md).

## Features

The web and desktop interfaces support:

- CSV import with configurable delimiter, encoding and missing-value policy.
- An interpreted preview of up to 100 rows before analysis.
- Dataset summaries and completeness scores.
- Searchable column cards and filters by column type.
- Numeric, text, date, datetime and boolean statistics.
- Paginated value frequencies, with missing values reported separately.
- Manual column types and restoration of automatic detection.
- Bar, line and scatter charts; Windows has no fixed series-count cap, while web/API support up to five series.
- JSON export of completed analyses, including manual types and import options.
- English and Spanish interfaces.

The desktop also exports charts as PNG and SVG. Both interfaces support original
line values, median/sample standard deviation, grouping, UTC calendar intervals,
independent measure scales, zoom/pan and light/dark/system themes. Desktop keeps
these operations local and preserves chart state across theme changes.
See [current status and validation](CURRENT.md) for scope and remaining checks.
The CLI prints a dataset profile to the
terminal; it does not expose all interactive features.

Type changes affect interpretation without modifying the source CSV. Failed
conversions preserve the previous analysis.

## Quick start with Docker

Requirements: Docker Engine and Docker Compose. On Windows, Docker Desktop must
use Linux containers.

From the repository root:

```powershell
docker compose up --build -d
```

| Service | Default address |
| --- | --- |
| Web interface | http://localhost:8080 |
| API health | http://localhost:8000/health |
| Effective API limits | http://localhost:8000/api/limits |

Inspect services and logs:

```powershell
docker compose ps
docker compose logs -f
```

Pressing `Ctrl+C` stops following logs while containers continue running.
To stop and remove the containers and their networks:

```powershell
docker compose down
```

You can also start the services independently:

```powershell
docker compose up --build -d api
docker compose up --build -d frontend
```

### Web architecture

- FastAPI serves `/health` and `/api/*`.
- Nginx serves the compiled React application and static assets.
- The browser sends requests directly to a configurable API URL.
- CORS allows explicitly configured browser origins.
- The containers use separate networks and can run independently.

The API does not require Node, React or a frontend build. It does not serve the
web application at `/`; that route returns 404. The frontend can load while the
API is unavailable, but CSV operations require a reachable API.

## Run from source

Development uses Python 3.13, uv, Node.js 22, npm and Git. Examples below use
PowerShell. On other shells, use `npm` instead of `npm.cmd` and adapt environment
variable syntax.

```powershell
git clone https://github.com/adribarrios05/data-inspector.git analytio
cd analytio
```

Install dependencies for the interface you need:

```powershell
# CLI
uv sync --locked

# API
uv sync --locked --extra api

# Desktop
uv sync --locked --extra desktop

# Complete Python development environment
uv sync --locked --extra api --extra desktop
```

These commands include development dependencies by default. For an API runtime
without development tools:

```powershell
uv sync --locked --no-dev --extra api
```

Python dependencies are locked in `uv.lock`. Frontend dependencies are locked in
`frontend/package-lock.json` and installed with `npm ci`.

### CLI

```powershell
uv run --locked analytio dataset.csv
uv run --locked analytio dataset.csv --separator ";" --encoding cp1252
uv run --locked analytio dataset.csv --null-policy empty_only
```

### Desktop

```powershell
uv run --locked --extra desktop analytio-desktop --language en
uv run --locked --extra desktop analytio-desktop --language es
```

Desktop processing is local and does not require the API. Once analysis finishes,
the application retains the original CSV bytes and interpreted rows in memory.
Frequencies, charts and type changes use this session even if the source file is
moved, modified or deleted. Import the file again to analyze updated content.

The session lasts while the application is open and requires memory for the
source bytes, interpreted data and temporary calculation copies.

### Web development

Run the API and Vite in separate terminals from the repository root.

Terminal 1:

```powershell
uv run --locked --extra api python -m analytio.presentation.api.dev
```

Terminal 2:

```powershell
npm.cmd --prefix frontend ci
npm.cmd --prefix frontend run dev
```

Open http://localhost:5173. The browser reads the API configuration from the versioned
configuration file `frontend/public/config.json`:

```json
{
  "apiBaseUrl": ""
}
```

The development launcher binds the API to loopback and supplies the two Vite
origins above when `ALLOWED_ORIGINS` is unset. An explicit value, including an
empty one, takes precedence. This avoids having to recreate a terminal's temporary
`$env:ALLOWED_ORIGINS` setting on each session. It watches only the Python package
for reloads. Stop an existing API on port 8000 before using this command.

The empty API base sends requests to `/api/...` on the page's own origin. Vite
proxies those paths to `http://127.0.0.1:8000` during development; the server-only
`DEV_API_PROXY_TARGET` in `frontend/.env.local` can override that target.
Production under one origin requires the reverse proxy to route `/api/` to
FastAPI. No tunnel hostname is embedded in the build. See the
[frontend deployment guide](frontend/README.md#lxc-nginx-and-cloudflare-quick-tunnel)
for LXC inspection, runtime config migration, build and deployment steps.
An explicit absolute `apiBaseUrl` remains supported for separately hosted APIs,
which still require an appropriate CORS configuration.

The configuration also accepts `requestTimeoutMs`, defaulting to 65000 ms.
Cancelling a browser request stops waiting for its response; it does not guarantee
immediate cancellation of server processing. The web interface lets you return to
the last completed analysis after a failed import.

Build the frontend:

```powershell
npm.cmd --prefix frontend run build
```

The output is generated in `frontend/dist` and does not require a running API.
Nginx serves this output in Docker. Chart code loads when the charts tab is first
opened. See the [frontend README](frontend/README.md) for additional instructions.

## CSV interpretation

| Missing-value policy | Behavior |
| --- | --- |
| `pandas` (default) | Uses standard Pandas missing-value tokens, including empty fields and tokens such as `NA` and `NULL`. |
| `empty_only` | Treats only empty fields as missing; preserves `NA`, `NULL` and whitespace values. |

Completely blank lines are skipped. Empty or duplicate column headers are
normalized by Pandas, retaining valid names verbatim. The preview and analysis
include `header_changes` (one-based position, original name and normalized name).
The interfaces display these changes and CLI writes warnings to stderr.
Records must have exactly as many fields as the header. Extra or missing fields
and malformed quoting are rejected rather than interpreted as an index, padded
or discarded. Quoted delimiters, escaped quotes and multiline fields are supported. Preview values are interpreted rather than displayed as
raw CSV text, and a preview does not validate the entire file.

Automatic detection can turn an identifier such as `001` into the number `1`.
Choose **Text** for identifiers to preserve their original lexical values.
The missing-value policy applies to analysis, frequencies and charts, and is
retained during manual type changes.

Automatic detection preserves long integers and missing cells without an
intermediate floating-point conversion, including signed Int64 boundaries.
Columns containing long integer tokens stay as text when their mixed values or
range cannot be represented exactly as Int64/UInt64. Manual Text preserves the
original spelling; manual Float remains an explicit approximate interpretation.

Manual conversions are strict. An incompatible value anywhere in the dataset,
including beyond the preview, can reject a conversion without replacing the last
successful analysis.

## Statistics and completeness

Common statistics include valid, missing and unique values, plus the most and
least frequent values. Type-specific statistics include:

- **Numeric:** mean, median, standard deviation, extrema, quartiles, 95th
  percentile, interquartile range and outliers.
- **Text:** lengths, words, empty strings, whitespace, numbers and symbols.
- **Dates:** extrema, temporal range, days with data, gaps and distributions.
- **Booleans:** counts, percentages, mode, ratio and entropy.

Frequency percentages use all rows as their denominator. Missing values are
reported separately. Values are ordered by frequency; ties retain encounter
order. Text and boolean metrics use non-null values. Dates are presented in UTC.
A dash in the interface indicates an undefined result.

The global score measures completeness only:

```text
completeness = (1 - missing cells / total cells) * 100
```

A high score does not establish correctness, business validity or absence of bias.

### Numeric precision

Integer statistics and original chart coordinates use JSON numbers within the
inclusive range `[-9007199254740991, 9007199254740991]`. Outside that range they
use exact decimal strings, such as `"9007199254740993"`. API consumers must accept
both representations and preserve these strings without coercing them to
JavaScript numbers. Frequency values are text.

Means, quantiles and floating-point aggregations remain approximate. Chart
renderers preserve original labels and tooltips while projecting coordinates for
display. The web renderer uses differences from an exact integer origin to
distinguish nearby large integers. Extremely wide ranges remain subject to
floating-point and screen resolution.

## Charts and exports

Charts use all admitted rows with the current delimiter, encoding, missing-value
policy and manual types.

| Chart | X axis | Aggregations |
| --- | --- | --- |
| Bar | Categories | `count`, `sum`, `mean`, `min`, `max` |
| Line | Numeric or temporal, sorted | `count`, `sum`, `mean`, `min`, `max` |
| Scatter | Numeric | `none`; Y must also be numeric |

Web/API charts support one to five series. Windows accepts any positive number
of series, including all series produced by grouping; usable capacity depends on
the data, rendering and available memory. `count` without a column counts rows; with a
column it counts non-null values. Rows with null X values are excluded and their
count is reported.

The API accepts `max_points` from 1 to 5000, defaulting to 1000. Bars reject
excessive groups instead of silently truncating them. Scatter charts sample
points and report the original count. Lines retain every ordered aggregate
within processing time and resource budgets, using continuous numeric/time axes and
no point markers. Null values break the line. Numeric aggregations use
floating-point arithmetic rather than guaranteeing exact integer arithmetic.

Appearance changes do not require another calculation. The web preserves the
last valid chart on errors and discards obsolete responses.

JSON export saves the completed profile, not the entire CSV or chart configuration:
summary, column statistics, frequency extrema, up to 100 sample rows, header changes, manual
types, `schema_version: 1` and `import_options`. Import options record the
delimiter, encoding and missing-value policy; they can be null for profiles
without CSV metadata. Desktop export does not reread the source file.

Exports can contain sensitive values in samples and frequency extrema. Review
them before sharing. Web and desktop charts can also be exported as PNG and SVG.

## HTTP API

| Method and route | Purpose |
| --- | --- |
| `GET /health` | Check that the service responds |
| `GET /api/limits` | Read effective CSV processing limits |
| `POST /api/previews` | Preview up to 100 interpreted rows |
| `POST /api/analyses` | Profile the complete uploaded CSV within operational budgets |
| `POST /api/frequencies` | Query paginated frequencies for one column |
| `POST /api/charts` | Calculate chart data within processing limits |

CSV endpoints accept multipart form data:

| Field | Usage |
| --- | --- |
| `file` | Required CSV file |
| `separator` | Single-character delimiter; defaults to `,` |
| `encoding` | Defaults to `utf-8` |
| `null_policy` | `pandas` or `empty_only`; defaults to `pandas` |
| `column_types` | JSON object of manual types for analysis, frequencies and charts |
| `column` | Required column name for frequencies |
| `offset` | Frequency offset; defaults to `0` |
| `limit` | Frequency page size from 1 to 100; defaults to `50` |
| `chart` | Required JSON configuration: `type`, `x`, `series` and optional `max_points` |

Example `column_types` value:

```json
{
  "id": "text",
  "age": "integer"
}
```

Supported manual types are `integer`, `float`, `text`, `boolean`, `date` and
`datetime`. Omit a column from the object to restore automatic detection.
Supported API encodings are `utf-8`, `utf-8-sig`, `cp1252`, `latin-1` and `utf-16`.

Each CSV operation uploads the file again. The API does not provide persistent
file identifiers or dataset history; temporary files are removed after processing.
Desktop processing is local, whereas web preview already uploads the CSV.

Interactive API documentation is disabled by default. Enable it for development:

```powershell
$env:ENABLE_API_DOCS = "true"
uv run --locked --extra api uvicorn analytio.presentation.api.main:app `
    --host 127.0.0.1 --port 8000 --reload
```

Open http://127.0.0.1:8000/docs. This is a runtime API route.

## Configuration and resource limits

The API reads process environment variables at startup. It does not automatically
load `.env` or `.env.example`. Use the versioned [.env.example](.env.example) as a
reference for names and defaults. The local development launcher documented above
supplies Vite origins only; it does not load these files either. For deployment,
continue to run `analytio.presentation.api.main:app` with explicit origins.
An empty origin list emits a diagnostic warning and disables cross-origin browser
access, even though the underlying endpoint can still respond with HTTP 200.

| Variable | Default |
| --- | --- |
| `MAX_UPLOAD_BYTES` | 20971520 (20 MiB) |
| `MAX_REQUEST_BYTES` | 22020096 (21 MiB, including multipart overhead) |
| `MAX_ANALYSIS_SECONDS` | 30 |
| `MAX_UPLOAD_SECONDS` | 30 |
| `MAX_BUFFERED_REQUESTS` | 4 |
| `RATE_LIMIT_REQUESTS` | 20 per client IP per window |
| `RATE_LIMIT_GLOBAL_REQUESTS` | 60 per server process per window |
| `RATE_LIMIT_WINDOW_SECONDS` | 60 |
| `ALLOWED_ORIGINS` | Empty unless configured |
| `ENABLE_API_DOCS` | `false` |

CSV files up to and including 20 MiB are admitted by default; there are no
row, column or cell admission limits. This does not guarantee that every admitted
file can be processed within the memory and time budgets. Preview remains a
sample of up to 100 rows, frequency pages contain up to 100 values, and chart
series/visualization budgets remain in effect.

`GET /api/limits` returns `upload_bytes`, `processing_seconds` and
`upload_seconds`; the former `rows`, `columns` and `cells` fields are removed.
Deploy matching API and frontend versions together. `MAX_REQUEST_BYTES` protects
the multipart body, not the dataset shape; its default is the upload budget plus
1 MiB (21 MiB with the default file size). Compose forwards both size settings.

Structural errors return 422 with `csv_structure_mismatch`, a logical record
number (header is record 1), its ending physical line, and expected/actual field
counts. Invalid quoting returns `invalid_csv_syntax`. Cell values are not included
in these errors. A preview validates only the sampled records; full operations
validate the complete input. `header_changes` is additive to schema version 1.

All four CSV endpoints share traffic and body limits. Upload reception has a time
budget and bounded admission. Preview parsing runs in an isolated process;
preliminary parsing and calculation share the processing time budget. One CSV
operation is processed at a time per server process. Increasing workers or
replicas requires reviewing these per-process controls.

Errors include 408 for upload timeout, 413 for excessive body/file size, 429 for
rate limiting, 503 for saturation and 504 for processing timeout. Allowed browser
origins receive CORS headers on early rejections, and can read `Retry-After`.
CORS is not authentication.

[Compose](compose.yaml) configures the API with one CPU, 2 GiB of memory, 128
processes, a read-only root filesystem and a 192 MiB temporary filesystem. These
are configured budgets, not measured production capacity. Its healthcheck proves
that the API responds, not that a dataset can be processed successfully.

Compose uses `.env` for substitutions but forwards only variables explicitly
listed under each service's `environment`. Adding a setting to `.env` alone does
not forward it to the API. `API_PORT` and `FRONTEND_PORT` control published ports.
`API_BASE_URL` must be reachable by the browser, not a Docker-internal service name.

## Windows packaging

Build on Windows x64 with the locked desktop dependencies and an existing Inno
Setup compiler:

```powershell
uv sync --locked --extra desktop
uv run --no-sync python scripts/build_windows.py
```

Each build creates a new directory under `dist/windows/` containing the bundle,
installer, frozen-app smoke report and manifest with version, source hashes and
artifact checksums. Sources are checked for changes during the build. A normal
development build may use a dirty checkout; the manifest records that fact.

`--bundle-only` builds without an installer. `--release` requires a clean checkout
and an active `LICENSE` file. The script does not install a compiler, create
commits, sign binaries or publish a release.

See the [Windows packaging README](packaging/README.md) for compiler options and
installed-app lifecycle checks. Public distribution still requires validation on
a clean Windows machine, normal GUI use, upgrades, uninstall and high-DPI displays.

## Development checks

With Python dependencies installed:

```powershell
$env:QT_QPA_PLATFORM = "offscreen"
uv run --no-sync ruff check src tests scripts packaging
uv run --no-sync pytest -q -p no:cacheprovider
uv build
```

With frontend dependencies installed:

```powershell
npm.cmd --prefix frontend run lint
npm.cmd --prefix frontend test
npm.cmd --prefix frontend run build

# Use an existing Chrome installation for local browser tests.
$env:PLAYWRIGHT_CHANNEL = "chrome"
npm.cmd --prefix frontend run test:e2e
```

Playwright exercises the real API and compiled React application in English and
Spanish. Its harness starts and stops its own servers, refuses to reuse existing
ones, and does not install dependencies. `E2E_PYTHON` can select an existing Python
interpreter; otherwise it uses the project virtual environment or Python on PATH.

Check configuration and whitespace:

```powershell
docker compose config --quiet
git diff --check
```

Validate and fingerprint a development candidate, or measure synthetic CSV operations:

```powershell
uv run --no-sync python scripts/verify_release.py --allow-dirty
uv run --no-sync python scripts/measure_csv.py `
    --output build/csv-measurements.json --repetitions 3
```

Candidate validation records actual check exits and source hashes. A dirty
checkout is development validation, not a reproducible Git release. Resource
measurements sample memory and report their limitations; they do not establish
server capacity.

The [CI workflow](.github/workflows/ci.yml) defines Windows Python/desktop checks,
Windows bundle and installed-app tests, Chromium web tests on Ubuntu and Docker
service checks. Pushes or workflow dispatches on `main` can trigger homelab
deployment after the required jobs pass, provided the self-hosted runner is
available. Repository configuration does not certify a successful remote run.

## Project structure

```text
src/analytio/
|-- application/       Use cases and application contracts
|-- domain/            Entities, types and ports
|-- infrastructure/    Readers, conversions, statistics and serialization
`-- presentation/
    |-- api/           HTTP API
    |-- cli/           Terminal output
    `-- desktop/       Desktop interface

frontend/              React application and Nginx image
tests/                 Python tests
scripts/               Validation, measurements and build tools
packaging/             Desktop bundle and installer configuration
```

The domain does not depend on Pandas. Data processing lives in infrastructure;
presentation adapters handle interaction, execution and rendering.

## Current limitations

- Active user workflows support CSV. Excel and Parquet are not integrated.
- Full analysis loads data into memory; desktop sessions retain additional data.
- Automatic date detection focuses on ISO 8601 formats.
- There are no accounts or persistent analysis history.
- Sensitive-data detection is not an active dataset feature.
- Web operations resend the CSV; cancelling HTTP does not immediately stop work.
- Desktop cancellation is cooperative where available; closing can wait for work.
- Application limits do not guarantee a fixed memory peak or multiuser capacity.
- Remote operation, rollback, clean-machine distribution and broader accessibility
  and browser coverage require separate validation.

## Public documentation

- [Frontend setup and tests](frontend/README.md)
- [Windows packaging](packaging/README.md)
- [Desktop translations](src/analytio/presentation/desktop/resources/translations/README.md)

## License

No active `LICENSE` file is currently included in the repository. Distribution
and usage terms have not been finalized. Public source availability alone should
not be treated as a published open-source license.
