mirror of
https://github.com/penpot/penpot-docs.git
synced 2024-08-04 09:58:54 +00:00
📚 Add architecture and data model
This commit is contained in:
Andrés Moya
parent
004af75b03
commit
21a40c50cc
@ -1,119 +0,0 @@
|
||||
---
|
||||
title: 3. Architecture
|
||||
---
|
||||
|
||||
# Architecture
|
||||
|
||||
This section gives an overall structure of the system.
|
||||
|
||||
Penpot has the architecture of a typical SPA. There is a frontend application,
|
||||
written in ClojureScript and using React framework, and served from a static
|
||||
web server. It talks to a backend application, that persists data on a
|
||||
PosgreSQL database.
|
||||
|
||||
The backend is written in Clojure, so front and back can share code and data
|
||||
structures without problem. Then, the code is compiled into JVM bytecode and
|
||||
run in a JVM environment.
|
||||
|
||||
There are some additional components, explained below.
|
||||
|
||||
@startuml C4_Elements
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
!include DEVICONS/java.puml
|
||||
!include DEVICONS/clojure.puml
|
||||
!include DEVICONS/postgresql.puml
|
||||
!include DEVICONS/redis.puml
|
||||
!include DEVICONS/chrome.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Person(user, "User")
|
||||
System_Boundary(frontend, "Frontend") {
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
Container(worker, "Worker", "Web worker")
|
||||
}
|
||||
|
||||
System_Boundary(backend, "Backend") {
|
||||
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
|
||||
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
|
||||
ContainerDb(redis, "Broker", "Redis", "", "redis")
|
||||
Container(exporter, "Exporter", "ClojureScript / nodejs", "", "clojure")
|
||||
Container(browser, "Headless browser", "Chrome", "", "chrome")
|
||||
}
|
||||
|
||||
Rel(user, frontend_app, "Uses", "HTTPS")
|
||||
BiRel_L(frontend_app, worker, "Works with")
|
||||
BiRel(frontend_app, backend_app, "Open", "websocket")
|
||||
Rel(frontend_app, backend_app, "Uses", "RPC API")
|
||||
Rel(backend_app, db, "Uses", "SQL")
|
||||
Rel(redis, backend_app, "Subscribes", "pub/sub")
|
||||
Rel(backend_app, redis, "Notifies", "pub/sub")
|
||||
Rel(frontend_app, exporter, "Uses", "HTTPS")
|
||||
Rel(exporter, browser, "Uses", "puppeteer")
|
||||
Rel(browser, frontend_app, "Uses", "HTTPS")
|
||||
|
||||
@enduml
|
||||
|
||||
## Frontend app
|
||||
|
||||
The main application, with the user interface and the presentation logic.
|
||||
|
||||
To talk with backend, it uses a custom RPC-style API: some functions in the
|
||||
backend are exposed through an HTTP server. When the front wants to execute a
|
||||
query or data mutation, it sends a HTTP request, containing the name of the
|
||||
function to execute, and the ascii-encoded arguments. The resulting data is
|
||||
also encoded and returned. This way we don't need any data type conversion,
|
||||
besides the transport encoding, as there is Clojure at both ends.
|
||||
|
||||
When the user opens any file, a persistent websocket is opened with the backend
|
||||
and associated to the file id. It is used to send presence events, such as
|
||||
connection, disconnection and mouse movements. And also to receive changes made
|
||||
by other users that are editing the same file, so it may be updated in real
|
||||
time.
|
||||
|
||||
## Worker
|
||||
|
||||
Some operations are costly to make in real time, so we leave them to be
|
||||
executed asynchronously in a web worker. This way they don't impact the user
|
||||
experience. Some of these operations are generating file thumbnails for the
|
||||
dashboard and maintaining some geometric indexes to speed up snap points while
|
||||
drawing.
|
||||
|
||||
## Backend app
|
||||
|
||||
This app is in charge of CRUD of data, integrity validation and persistence
|
||||
into a database and also into a file system for media attachments.
|
||||
|
||||
To handle deletions it uses a garbage collector mechanism: no object in the
|
||||
database is deleted instantly. Instead, a field `deleted_at` is set with the
|
||||
date and time of the deletion, and every query ignores db rows that have this
|
||||
field set. Then, an async task that runs periodically, locates rows whose
|
||||
deletion date is older than a given threshold and permanently deletes them.
|
||||
|
||||
For this, and other possibly slow tasks, there is an internal async tasks
|
||||
worker, that may be used to queue tasks to be scheduled and executed when the
|
||||
backend is idle. Other tasks are email sending, collecting data for telemetry
|
||||
and detecting unused media attachment, for removing them from the file storage.
|
||||
|
||||
## PubSub
|
||||
|
||||
To manage subscriptions to a file, to be notified of changes, we use a redis
|
||||
server as a pub/sub broker. Whenever a user visits a file and opens a
|
||||
websocket, the backend creates a subscription in redis, with a topic that has
|
||||
the id of the file. If the user sends any change to the file, backend sends a
|
||||
notification to this topic, that is received by all subscribers. Then the
|
||||
notification is retrieved and send to the user via the websocket.
|
||||
|
||||
## Exporter
|
||||
|
||||
When exporting file contents to a file, we want the result to be exactly the
|
||||
same as the user sees in screen. To achieve this, we use a headless browser
|
||||
installed in the backend host, and controled via puppeteer automation. The
|
||||
browser loads the frontend app from the static webserver, and executes it like
|
||||
a normal user browser. It visits a special endpoint that renders one shape
|
||||
inside a file. Then, if takes a screenshot if we are exporting to a bitmap
|
||||
image, or extract the svg from the DOM if we want a vectorial export, and write
|
||||
it to a file that the user can download.
|
||||
|
||||
136
technical-guide/architecture/backend.md
Normal file
136
technical-guide/architecture/backend.md
Normal file
@ -0,0 +1,136 @@
|
||||
---
|
||||
title: 3.2. Backend app
|
||||
---
|
||||
|
||||
# Backend app
|
||||
|
||||
This app is in charge of CRUD of data, integrity validation and persistence
|
||||
into a database and also into a file system for media attachments.
|
||||
|
||||
To handle deletions it uses a garbage collector mechanism: no object in the
|
||||
database is deleted instantly. Instead, a field `deleted_at` is set with the
|
||||
date and time of the deletion, and every query ignores db rows that have this
|
||||
field set. Then, an async task that runs periodically, locates rows whose
|
||||
deletion date is older than a given threshold and permanently deletes them.
|
||||
|
||||
For this, and other possibly slow tasks, there is an internal async tasks
|
||||
worker, that may be used to queue tasks to be scheduled and executed when the
|
||||
backend is idle. Other tasks are email sending, collecting data for telemetry
|
||||
and detecting unused media attachment, for removing them from the file storage.
|
||||
|
||||
# Backend structure
|
||||
|
||||
Penpot backend app code resides under `backend/src/app` path in the main repository.
|
||||
|
||||
@startuml BackendGeneral
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
!include DEVICONS/java.puml
|
||||
!include DEVICONS/clojure.puml
|
||||
!include DEVICONS/postgresql.puml
|
||||
!include DEVICONS/redis.puml
|
||||
!include DEVICONS/chrome.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
|
||||
System_Boundary(backend, "Backend") {
|
||||
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
|
||||
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
|
||||
ContainerDb(redis, "Broker", "Redis", "", "redis")
|
||||
}
|
||||
|
||||
BiRel(frontend_app, backend_app, "Open", "websocket")
|
||||
Rel(frontend_app, backend_app, "Uses", "RPC API")
|
||||
Rel(backend_app, db, "Uses", "SQL")
|
||||
Rel(redis, backend_app, "Subscribes", "pub/sub")
|
||||
Rel(backend_app, redis, "Notifies", "pub/sub")
|
||||
|
||||
@enduml
|
||||
|
||||
```
|
||||
▾ backend/src/app/
|
||||
▸ cli/
|
||||
▸ http/
|
||||
▸ migrations/
|
||||
▸ rpc/
|
||||
▸ setup/
|
||||
▸ srepl/
|
||||
▸ util/
|
||||
▸ tasks/
|
||||
main.clj
|
||||
config.clj
|
||||
http.clj
|
||||
metrics.clj
|
||||
migrations.clj
|
||||
notifications.clj
|
||||
rpc.clj
|
||||
setup.clj
|
||||
srepl.clj
|
||||
worker.clj
|
||||
...
|
||||
```
|
||||
|
||||
* `main.clj` defines the app global settings and the main entry point of the
|
||||
application, served by a JVM.
|
||||
* `config.clj` defines of the configuration options read from linux
|
||||
environment.
|
||||
* `http` contains the HTTP server and the backend routes list.
|
||||
* `migrations` contains the SQL scripts that define the database schema, in
|
||||
the form of a sequence of migrations.
|
||||
* `rpc` is the main module to handle the RPC API calls.
|
||||
* `notifications.clj` is the main module that manages the websocket. It allows
|
||||
clients to subscribe to open files, intercepts update RPC calls and notify
|
||||
them to all subscribers of the file.
|
||||
* `setup` initializes the environment (loads config variables, sets up the
|
||||
database, executes migrations, loads initial data, etc).
|
||||
* `srepl` sets up an interactive REPL shell, with some useful commands to be
|
||||
used to debug a running instance.
|
||||
* `cli` sets a command-line interface, with some more maintenance commands.
|
||||
* `metrics.clj` has some interceptors that watches RPC calls, calculate
|
||||
statistics and other metrics, and send them to external systems to store and
|
||||
analyze.
|
||||
* `worker.clj` and `tasks` define some async tasks that are executed in
|
||||
parallel to the main http server (using java threads), and scheduled in a
|
||||
cron-like table. They are useful to do some garbage collection, data packing
|
||||
and similar periodic maintenance tasks.
|
||||
* `db.clj`, `emails.clj`, `media.clj`, `msgbus.clj`, `storage.clj`,
|
||||
`rlimits.clj` are general libraries to use I/O resources (SQL database,
|
||||
send emails, handle multimedia objects, use REDIS messages, external file
|
||||
storage and semaphores).
|
||||
* `util/` has a collection of generic utility functions.
|
||||
|
||||
## RPC calls
|
||||
|
||||
The RPC (Remote Procedure Call) subsystem consists of a mechanism that allows
|
||||
to expose clojure functions as an HTTP endpoint. We take advantage of being
|
||||
using Clojure at both front and back ends, to avoid needing complex data
|
||||
conversions.
|
||||
|
||||
1. Frontend initiates a "query" or "mutation" call to `:xxx` method, and
|
||||
passes a Clojure object as params.
|
||||
2. Params are string-encoded using
|
||||
[transit](https://github.com/cognitect/transit-clj), a format similar to
|
||||
JSON but more powerful.
|
||||
3. The call is mapped to `<backend-host>/api/rpc/query/xxx` or
|
||||
`<backend-host>/api/rpc/mutation/xxx`.
|
||||
4. The `rpc` module receives the call, decode the parameters and executes the
|
||||
corresponding method inside `src/app/rpc/queries/` or `src/app/rpc/mutations/`.
|
||||
We have created a `defmethod` macro to declare an RPC method and its
|
||||
parameter specs.
|
||||
5. The result value is also transit-encoded and returned to the frontend.
|
||||
|
||||
This way, frontend can execute backend calls like it was calling an async function,
|
||||
with all the power of Clojure data structures.
|
||||
|
||||
## PubSub
|
||||
|
||||
To manage subscriptions to a file, to be notified of changes, we use a redis
|
||||
server as a pub/sub broker. Whenever a user visits a file and opens a
|
||||
websocket, the backend creates a subscription in redis, with a topic that has
|
||||
the id of the file. If the user sends any change to the file, backend sends a
|
||||
notification to this topic, that is received by all subscribers. Then the
|
||||
notification is retrieved and send to the user via the websocket.
|
||||
|
||||
84
technical-guide/architecture/common.md
Normal file
84
technical-guide/architecture/common.md
Normal file
@ -0,0 +1,84 @@
|
||||
---
|
||||
title: 3.4. Common code
|
||||
---
|
||||
|
||||
# Common code
|
||||
|
||||
In penpot, we take advantage of using the same language in frontend and
|
||||
backend, to have a bunch of shared code.
|
||||
|
||||
Sometimes, we use conditional compilation, for small chunks of code that
|
||||
are different in a Clojure+Java or ClojureScript+JS environments. We use
|
||||
the `#?` construct, like this, for example:
|
||||
|
||||
```clojure
|
||||
(defn ordered-set?
|
||||
[o]
|
||||
#?(:cljs (instance? lks/LinkedSet o)
|
||||
:clj (instance? LinkedSet o)))
|
||||
```
|
||||
|
||||
```
|
||||
▾ common/src/app/common/
|
||||
▸ geom/
|
||||
▸ pages/
|
||||
▸ path/
|
||||
▸ types/
|
||||
...
|
||||
```
|
||||
|
||||
Some of the modules need some refactoring, to organize them more cleanly.
|
||||
|
||||
# Data model and business logic
|
||||
|
||||
* **geom** contains functions to manage 2D geometric entities.
|
||||
- **point** defines the 2D Point type and many geometric transformations.
|
||||
- **matrix** defines the [2D transformation
|
||||
matrix](https://www.alanzucconi.com/2016/02/10/tranfsormation-matrix/)
|
||||
type and its operations.
|
||||
- **shapes** manages shapes as a collection of points with a bounding
|
||||
rectangle.
|
||||
* **path** contains functions to manage SVG paths, transform them and also
|
||||
convert other types of shapes into paths.
|
||||
* **pages** contains the definition of the [Penpot data model](./model.md) and
|
||||
the conceptual business logic (transformations of the model entities,
|
||||
independent of the user interface or data storage).
|
||||
- **spec** has the definitions of data structures of files and shapes, and
|
||||
also of the transformation operations in **changes** module. Uses [Clojure
|
||||
spec](https://github.com/clojure/spec.alpha) to define the structure and
|
||||
validators.
|
||||
- **init** defines the default content of files, pages and shapes.
|
||||
- **helpers** are some functions to help manipulating the data structures.
|
||||
- **migrations** is in charge to manage the evolution of the data model
|
||||
structure over time. It contains a function that gets a file data
|
||||
content, identifies its version, and applies the needed migrations. Much
|
||||
like the SQL database migrations scripts.
|
||||
- **changes** and **changes_builder** define a set of transactional
|
||||
operations, that receive a file data content, and perform a semantic
|
||||
operation following the business logic (add a page or a shape, change a
|
||||
shape attribute, modify some file asset, etc.).
|
||||
* **types** we are currently in process of refactoring **pages** module, to
|
||||
organize it in a way more compliant of [Abstract Data
|
||||
Types](https://en.wikipedia.org/wiki/Abstract_data_type) paradigm. We are
|
||||
approaching the process incrementally, rewriting one module each time, as
|
||||
needed.
|
||||
|
||||
# Utilities
|
||||
|
||||
The main ones are:
|
||||
|
||||
* **data** basic data structures and utility functions that could be added to
|
||||
Clojure standard library.
|
||||
* **math** some mathematic functions that could also be standard.
|
||||
* **file_builder** functions to parse the content of a `.penpot` exported file
|
||||
and build a File data structure from it.
|
||||
* **logging** functions to generate traces for debugging and usage analysis.
|
||||
* **text** an adapter layer over the [DraftJS editor](https://draftjs.org) that
|
||||
we use to edit text shapes in workspace.
|
||||
* **transit** functions to encode/decode Clojure objects into
|
||||
[transit](https://github.com/cognitect/transit-clj), a format similar to JSON
|
||||
but more powerful.
|
||||
* **uuid** functions to generate [Universally Unique Identifiers
|
||||
(UUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier), used
|
||||
over all Penpot models to have identifiers for objects that are practically
|
||||
ensured to be unique, without having a central control.
|
||||
68
technical-guide/architecture/exporter.md
Normal file
68
technical-guide/architecture/exporter.md
Normal file
@ -0,0 +1,68 @@
|
||||
---
|
||||
title: 3.3. Exporter app
|
||||
---
|
||||
|
||||
# Exporter app
|
||||
|
||||
When exporting file contents to a file, we want the result to be exactly the
|
||||
same as the user sees in screen. To achieve this, we use a headless browser
|
||||
installed in the backend host, and controled via puppeteer automation. The
|
||||
browser loads the frontend app from the static webserver, and executes it like
|
||||
a normal user browser. It visits a special endpoint that renders one shape
|
||||
inside a file. Then, if takes a screenshot if we are exporting to a bitmap
|
||||
image, or extract the svg from the DOM if we want a vectorial export, and write
|
||||
it to a file that the user can download.
|
||||
|
||||
# Exporter structure
|
||||
|
||||
Penpot exporter app code resides under `exporter/src/app` path in the main repository.
|
||||
|
||||
@startuml Exporter
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
!include DEVICONS/clojure.puml
|
||||
!include DEVICONS/chrome.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
|
||||
System_Boundary(backend, "Backend") {
|
||||
Container(exporter, "Exporter", "ClojureScript / nodejs", "", "clojure")
|
||||
Container(browser, "Headless browser", "Chrome", "", "chrome")
|
||||
}
|
||||
|
||||
Rel_D(frontend_app, exporter, "Uses", "HTTPS")
|
||||
Rel_R(exporter, browser, "Uses", "puppeteer")
|
||||
Rel_U(browser, frontend_app, "Uses", "HTTPS")
|
||||
|
||||
@enduml
|
||||
|
||||
```
|
||||
▾ exporter/src/app/
|
||||
▸ http/
|
||||
▸ renderer/
|
||||
▸ util/
|
||||
core.cljs
|
||||
http.cljs
|
||||
browser.cljs
|
||||
config.cljs
|
||||
```
|
||||
|
||||
## Exporter namespaces
|
||||
|
||||
* **core** has the setup and run functions of the nodejs app.
|
||||
|
||||
* **http** exposes a basic http server, with endpoints to export a shape or a
|
||||
file.
|
||||
|
||||
* **browser** has functions to control a local Chromium browser via
|
||||
[puppeteer](https://puppeteer.github.io/puppeteer).
|
||||
|
||||
* **renderer** has functions to tell the browser to render an object and make a
|
||||
screenshot, and then convert it to bitmap, pdf or svg as needed.
|
||||
|
||||
* **config** gets configuration settings from the linux environment.
|
||||
|
||||
* **util** has some generic utility functions.
|
||||
258
technical-guide/architecture/frontend.md
Normal file
258
technical-guide/architecture/frontend.md
Normal file
@ -0,0 +1,258 @@
|
||||
---
|
||||
title: 3.1. Frontend app
|
||||
---
|
||||
|
||||
# Frontend app
|
||||
|
||||
The main application, with the user interface and the presentation logic.
|
||||
|
||||
To talk with backend, it uses a custom RPC-style API: some functions in the
|
||||
backend are exposed through an HTTP server. When the front wants to execute a
|
||||
query or data mutation, it sends a HTTP request, containing the name of the
|
||||
function to execute, and the ascii-encoded arguments. The resulting data is
|
||||
also encoded and returned. This way we don't need any data type conversion,
|
||||
besides the transport encoding, as there is Clojure at both ends.
|
||||
|
||||
When the user opens any file, a persistent websocket is opened with the backend
|
||||
and associated to the file id. It is used to send presence events, such as
|
||||
connection, disconnection and mouse movements. And also to receive changes made
|
||||
by other users that are editing the same file, so it may be updated in real
|
||||
time.
|
||||
|
||||
# Frontend structure
|
||||
|
||||
Penpot frontend app code resides under `frontend/src/app` path in the main repository.
|
||||
|
||||
@startuml FrontendGeneral
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Person(user, "User")
|
||||
System_Boundary(frontend, "Frontend") {
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
Container(worker, "Worker", "Web worker")
|
||||
}
|
||||
|
||||
Rel(user, frontend_app, "Uses", "HTTPS")
|
||||
BiRel_L(frontend_app, worker, "Works with")
|
||||
|
||||
@enduml
|
||||
|
||||
```
|
||||
▾ frontend/src/app/
|
||||
▸ main/
|
||||
▸ util/
|
||||
▸ worker/
|
||||
main.cljs
|
||||
worker.cljs
|
||||
```
|
||||
|
||||
* `main.cljs` and `main/` contain the main frontend app, written in
|
||||
ClojureScript language and using React framework, wrapped in [rumext
|
||||
library](https://github.com/funcool/rumext).
|
||||
* `worker.cljs` and `worker/` contain the web worker, to make expensive
|
||||
calculations in background.
|
||||
* `util/` contains many generic utilities, non dependant on the user
|
||||
interface.
|
||||
|
||||
@startuml FrontendMain
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Component(ui, "ui", "main web component")
|
||||
Component(store, "store", "module")
|
||||
Component(refs, "refs", "module")
|
||||
Component(repo, "repo", "module")
|
||||
Component(streams, "streams", "module")
|
||||
Component(errors, "errors", "module")
|
||||
|
||||
Boundary(ui_namespaces, "ui namespaces") {
|
||||
Component(ui_auth, "auth", "web component")
|
||||
Component(ui_settings, "settings", "web component")
|
||||
Component(ui_dashboard, "dashboard", "web component")
|
||||
Component(ui_workspace, "workspace", "web component")
|
||||
Component(ui_viewer, "viewer", "web component")
|
||||
Component(ui_render, "render", "web component")
|
||||
Component(ui_exports, "exports", "web component")
|
||||
Component(ui_shapes, "shapes", "component library")
|
||||
Component(ui_components, "components", "component library")
|
||||
}
|
||||
|
||||
Boundary(data_namespaces, "data namespaces") {
|
||||
Component(data_common, "common", "events")
|
||||
Component(data_users, "users", "events")
|
||||
Component(data_dashboard, "dashboard", "events")
|
||||
Component(data_workspace, "workspace", "events")
|
||||
Component(data_viewer, "viewer", "events")
|
||||
Component(data_comments, "comments", "events")
|
||||
Component(data_fonts, "fonts", "events")
|
||||
Component(data_messages, "messages", "events")
|
||||
Component(data_modal, "modal", "events")
|
||||
Component(data_shortcuts, "shortcuts", "utilities")
|
||||
}
|
||||
|
||||
Lay_D(ui_exports, data_viewer)
|
||||
Lay_D(ui_settings, ui_components)
|
||||
Lay_D(data_viewer, data_common)
|
||||
Lay_D(data_fonts, data_messages)
|
||||
Lay_D(data_dashboard, data_modal)
|
||||
Lay_D(data_workspace, data_shortcuts)
|
||||
Lay_L(data_dashboard, data_fonts)
|
||||
Lay_L(data_workspace, data_comments)
|
||||
|
||||
Rel_Up(refs, store, "Watches")
|
||||
Rel_Up(streams, store, "Watches")
|
||||
|
||||
Rel(ui, ui_auth, "Routes")
|
||||
Rel(ui, ui_settings, "Routes")
|
||||
Rel(ui, ui_dashboard, "Routes")
|
||||
Rel(ui, ui_workspace, "Routes")
|
||||
Rel(ui, ui_viewer, "Routes")
|
||||
Rel(ui, ui_render, "Routes")
|
||||
|
||||
Rel(ui_render, ui_exports, "Uses")
|
||||
Rel(ui_workspace, ui_shapes, "Uses")
|
||||
Rel(ui_viewer, ui_shapes, "Uses")
|
||||
Rel_Right(ui_exports, ui_shapes, "Uses")
|
||||
|
||||
Rel(ui_auth, data_users, "Uses")
|
||||
Rel(ui_settings, data_users, "Uses")
|
||||
Rel(ui_dashboard, data_dashboard, "Uses")
|
||||
Rel(ui_dashboard, data_fonts, "Uses")
|
||||
Rel(ui_workspace, data_workspace, "Uses")
|
||||
Rel(ui_workspace, data_comments, "Uses")
|
||||
Rel(ui_viewer, data_viewer, "Uses")
|
||||
|
||||
@enduml
|
||||
|
||||
## General namespaces
|
||||
|
||||
* **store** contains the global state of the application. Uses an event loop
|
||||
paradigm, similar to Redux, with a global state object and a stream of events
|
||||
that modify it. Made with [potok library](https://funcool.github.io/potok).
|
||||
|
||||
* **refs** has the collection of references or lenses: RX streams that you can
|
||||
use to subscribe to parts of the global state, and be notified when they
|
||||
change.
|
||||
|
||||
* **streams** has some streams, derived from the main event stream, for keyboard
|
||||
and mouse events. Used mainly from the workspace viewport.
|
||||
|
||||
* **repo** contains the functions to make calls to backend.
|
||||
|
||||
* **errors** has functions with global error handlers, to manage exceptions or other
|
||||
kinds of errors in the ui or the data events, notify the user in a useful way,
|
||||
and allow to recover and continue working.
|
||||
|
||||
## UI namespaces
|
||||
|
||||
* **ui** is the root web component. It reads the current url and mounts the needed
|
||||
subcomponent depending on the route.
|
||||
|
||||
* **auth** has the web components for the login, register, password recover,
|
||||
etc. screens.
|
||||
|
||||
* **settings** has the web comonents for the user profile and settings screens.
|
||||
|
||||
* **dashboard** has the web components for the dashboard and its subsections.
|
||||
|
||||
* **workspace** has the web components for the file workspace and its subsections.
|
||||
|
||||
* **viewer** has the web components for the viewer and its subsections.
|
||||
|
||||
* **render** contain special web components to render one page or one specific
|
||||
shape, to be used in exports.
|
||||
|
||||
* **export** contain basic web components that display one shape or frame, to
|
||||
be used from exports render or else from dashboard and viewer thumbnails and
|
||||
other places.
|
||||
|
||||
* **shapes** is the basic collection of web components that convert all types of
|
||||
shapes in the corresponding svg elements, without adding any extra function.
|
||||
|
||||
* **components** a library of generic UI widgets, to be used as building blocks
|
||||
of penpot screens (text or numeric inputs, selects, forms, buttons...).
|
||||
|
||||
|
||||
## Data namespaces
|
||||
|
||||
* **users** has events to login and register, fetch the user profile and update it.
|
||||
|
||||
* **dashboard** has events to fetch and modify teams, projects and files.
|
||||
|
||||
* **fonts** has some extra events to manage uploaded fonts from dashboard.
|
||||
|
||||
* **workspace** has a lot of events to manage the current file and do all kinds of
|
||||
edits and updates.
|
||||
|
||||
* **comments** has some extra events to manage design comments.
|
||||
|
||||
* **viewer** has events to fetch a file contents to display, and manage the
|
||||
interactive behavior and hand-off.
|
||||
|
||||
* **common** has some events used from several places.
|
||||
|
||||
* **modal** has some events to show modal popup windows.
|
||||
|
||||
* **messages** has some events to show non-modal informative messages.
|
||||
|
||||
* **shortcuts** has some utility functions, used in other modules to setup the
|
||||
keyboard shortcuts.
|
||||
|
||||
|
||||
# Worker app
|
||||
|
||||
Some operations are costly to make in real time, so we leave them to be
|
||||
executed asynchronously in a web worker. This way they don't impact the user
|
||||
experience. Some of these operations are generating file thumbnails for the
|
||||
dashboard and maintaining some geometric indexes to speed up snap points while
|
||||
drawing.
|
||||
|
||||
@startuml FrontendWorker
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Component(worker, "worker", "worker entry point")
|
||||
|
||||
Boundary(worker_namespaces, "worker namespaces") {
|
||||
Component(thumbnails, "thumbnails", "worker methods")
|
||||
Component(snaps, "snaps", "worker methods")
|
||||
Component(selection, "selection", "worker methods")
|
||||
Component(impl, "impl", "worker methods")
|
||||
Component(import, "import", "worker methods")
|
||||
Component(export, "export", "worker methods")
|
||||
}
|
||||
|
||||
Rel(worker, thumbnails, "Uses")
|
||||
Rel(worker, impl, "Uses")
|
||||
Rel(worker, import, "Uses")
|
||||
Rel(worker, export, "Uses")
|
||||
Rel(impl, snaps, "Uses")
|
||||
Rel(impl, selection, "Uses")
|
||||
|
||||
@enduml
|
||||
|
||||
* **worker** contains the worker setup code and the global handler that receives
|
||||
requests from the main app, and process them.
|
||||
|
||||
* **thumbnails** has a method to generate the file thumbnails used in dashboard.
|
||||
|
||||
* **snaps** manages a distance index of shapes, and has a method to get
|
||||
other shapes near a given one, to be used in snaps while drawing.
|
||||
|
||||
* **selection** manages a geometric index of shapes, with methods to get what
|
||||
shapes are under the cursor at a given moment, for select.
|
||||
|
||||
* **impl** has a simple method to update all indexes in a page at once.
|
||||
|
||||
* **import** has a method to import a whole file from an external `.penpot` archive.
|
||||
|
||||
* **export** has a method to export a whole file to an external `.penpot` archive.
|
||||
|
||||
65
technical-guide/architecture/index.md
Normal file
65
technical-guide/architecture/index.md
Normal file
@ -0,0 +1,65 @@
|
||||
---
|
||||
title: 3. Architecture
|
||||
---
|
||||
|
||||
# Architecture
|
||||
|
||||
This section gives an overall structure of the system.
|
||||
|
||||
Penpot has the architecture of a typical SPA. There is a frontend application,
|
||||
written in ClojureScript and using React framework, and served from a static
|
||||
web server. It talks to a backend application, that persists data on a
|
||||
PosgreSQL database.
|
||||
|
||||
The backend is written in Clojure, so front and back can share code and data
|
||||
structures without problem. Then, the code is compiled into JVM bytecode and
|
||||
run in a JVM environment.
|
||||
|
||||
There are some additional components, explained in subsections.
|
||||
|
||||
@startuml C4_Elements
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
!include DEVICONS/java.puml
|
||||
!include DEVICONS/clojure.puml
|
||||
!include DEVICONS/postgresql.puml
|
||||
!include DEVICONS/redis.puml
|
||||
!include DEVICONS/chrome.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Person(user, "User")
|
||||
System_Boundary(frontend, "Frontend") {
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
Container(worker, "Worker", "Web worker")
|
||||
}
|
||||
|
||||
System_Boundary(backend, "Backend") {
|
||||
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
|
||||
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
|
||||
ContainerDb(redis, "Broker", "Redis", "", "redis")
|
||||
Container(exporter, "Exporter", "ClojureScript / nodejs", "", "clojure")
|
||||
Container(browser, "Headless browser", "Chrome", "", "chrome")
|
||||
}
|
||||
|
||||
Rel(user, frontend_app, "Uses", "HTTPS")
|
||||
BiRel_L(frontend_app, worker, "Works with")
|
||||
BiRel(frontend_app, backend_app, "Open", "websocket")
|
||||
Rel(frontend_app, backend_app, "Uses", "RPC API")
|
||||
Rel(backend_app, db, "Uses", "SQL")
|
||||
Rel(redis, backend_app, "Subscribes", "pub/sub")
|
||||
Rel(backend_app, redis, "Notifies", "pub/sub")
|
||||
Rel(frontend_app, exporter, "Uses", "HTTPS")
|
||||
Rel(exporter, browser, "Uses", "puppeteer")
|
||||
Rel(browser, frontend_app, "Uses", "HTTPS")
|
||||
|
||||
@enduml
|
||||
|
||||
See more at
|
||||
|
||||
* [Frontend app](/technical-guide/architecture/frontend/)
|
||||
* [Backend app](/technical-guide/architecture/backend/)
|
||||
* [Exporter app](/technical-guide/architecture/exporter/)
|
||||
* [Common code](/technical-guide/architecture/common/)
|
||||
|
||||
199
technical-guide/data-model/index.md
Normal file
199
technical-guide/data-model/index.md
Normal file
@ -0,0 +1,199 @@
|
||||
---
|
||||
title: 4. Data model
|
||||
---
|
||||
|
||||
# Penpot Data Model
|
||||
|
||||
This is the conceptual data model. The actual representations of those entities
|
||||
slightly differ, depending on the environment (frontend app, backend RPC calls
|
||||
or the SQL database, for example). But the concepts are always the same.
|
||||
|
||||
The diagrams use [basic UML notation with PlantUML](https://plantuml.com/en/class-diagram).
|
||||
|
||||
## Users, teams and projects
|
||||
|
||||
@startuml TeamModel
|
||||
|
||||
hide members
|
||||
|
||||
class Profile
|
||||
class Team
|
||||
class Project
|
||||
class File
|
||||
class StorageObject
|
||||
class CommentThread
|
||||
class Comment
|
||||
class ShareLink
|
||||
|
||||
Profile "*" -right- "*" Team
|
||||
Team *--> "*" Project
|
||||
Profile "*" -- "*" Project
|
||||
Project *--> "*" File
|
||||
Profile "*" -- "*" File
|
||||
File "*" <-- "*" File : libraries
|
||||
File *--> "*" StorageObject : media_objects
|
||||
File *--> "*" CommentThread : comment_threads
|
||||
CommentThread *--> "*" Comment
|
||||
File *--> "*" ShareLink : share_links
|
||||
|
||||
@enduml
|
||||
|
||||
A `Profile` holds the personal info of any user of the system. Users belongs to
|
||||
`Teams` and may create `Projects` inside them.
|
||||
|
||||
Inside the projects, there are `Files`. All users of a team may see the projects
|
||||
and files inside the team. Also, any project and file has at least one user that
|
||||
is the owner, but may have more relationships with users with other roles.
|
||||
|
||||
Files may use other files as shared `libraries`.
|
||||
|
||||
The main content of the file is in the "file data" attribute (see next section).
|
||||
But there are some objects that reside in separate entities:
|
||||
|
||||
* A `StorageObject` represents a file in an external storage, that is embedded
|
||||
into a file (currently images and SVG icons, but we may add other media
|
||||
types in the future).
|
||||
|
||||
* `CommentThreads`and `Comments` are the comments that any user may add to a
|
||||
file.
|
||||
|
||||
* A `ShareLink` contains a token, an URL and some permissions to share the file
|
||||
with external users.
|
||||
|
||||
## File data
|
||||
|
||||
@startuml FileModel
|
||||
|
||||
hide members
|
||||
|
||||
class File
|
||||
class Page
|
||||
class Component
|
||||
class Color
|
||||
class MediaItem
|
||||
class Typography
|
||||
|
||||
File *--> "*" Page : pages
|
||||
(File, Page) .. PagesList
|
||||
|
||||
File *--> "*" Component : components
|
||||
(File, Component) .. ComponentsList
|
||||
|
||||
File *--> "*" Color : colors
|
||||
(File, Color) .. ColorsList
|
||||
|
||||
File *--> "*" MediaItem : colors
|
||||
(File, MediaItem) .. MediaItemsList
|
||||
|
||||
File *--> "*" Typography : colors
|
||||
(File, Typography) .. TypographiesList
|
||||
|
||||
@enduml
|
||||
|
||||
The data attribute contains the `Pages` and the library assets in the file
|
||||
(`Components`, `MediaItems`, `Colors` and `Typographies`).
|
||||
|
||||
The lists of pages and assets are modelled also as entities because they have a
|
||||
lot of functions and business logic.
|
||||
|
||||
## Pages and components
|
||||
|
||||
@startuml PageModel
|
||||
|
||||
hide members
|
||||
|
||||
class Container
|
||||
class Page
|
||||
class Component
|
||||
class Shape
|
||||
|
||||
Container <|-left- Page
|
||||
Container <|-right- Component
|
||||
|
||||
Container *--> "*" Shape : objects
|
||||
(Container, Shape) .. ShapeTree
|
||||
|
||||
Shape <-- Shape : parent
|
||||
|
||||
@enduml
|
||||
|
||||
Both `Pages` and `Components` contains a tree of shapes, and share many
|
||||
functions and logic. So, we have modelled a `Container` entity, that is an
|
||||
abstraction that represents both a page or a component, to use it whenever we
|
||||
have code that fits the two.
|
||||
|
||||
A `ShapeTree` represents a set of shapes that are hierarchically related: the top
|
||||
frame contains top-level shapes (frames and other shapes). Frames and groups may
|
||||
contain any non frame shape.
|
||||
|
||||
## Shapes
|
||||
|
||||
@startuml ShapeModel
|
||||
|
||||
hide members
|
||||
|
||||
class Shape
|
||||
class Selrect
|
||||
class Transform
|
||||
class Constraints
|
||||
class Interactions
|
||||
class Fill
|
||||
class Stroke
|
||||
class Shadow
|
||||
class Blur
|
||||
class Font
|
||||
class Content
|
||||
class Exports
|
||||
|
||||
Shape o--> Selrect
|
||||
Shape o--> Transform
|
||||
Shape o--> Constraints
|
||||
Shape o--> Interactions
|
||||
Shape o--> Fill
|
||||
Shape o--> Stroke
|
||||
Shape o--> Shadow
|
||||
Shape o--> Blur
|
||||
Shape o--> Font
|
||||
Shape o--> Content
|
||||
Shape o--> Exports
|
||||
|
||||
Shape <-- Shape : parent
|
||||
|
||||
@enduml
|
||||
|
||||
A `Shape` is the most important entity of the model. Represents one of the
|
||||
[layers of our design](https://help.penpot.app/user-guide/layer-basics), and it
|
||||
corresponds with one SVG node, augmented with Penpot special features.
|
||||
|
||||
We have code to render a `Shape` into a SVG tag, with more or less additions
|
||||
depending on the environment (editable in the workspace, interactive in the
|
||||
viewer, minimal in the shape exporter or the handoff, or with metadata in the
|
||||
file export).
|
||||
|
||||
Also have code that imports any SVG file and convert elements back into shapes.
|
||||
If it's a SVG exported by Penpot, it reads the metadata to reconstruct the
|
||||
shapes exactly as they were. If not, it infers the atributes with a best effort
|
||||
approach.
|
||||
|
||||
In addition to the identifier ones (the id, the name and the type of element),
|
||||
a shape has a lot of attributes. We tend to group them in related clusters.
|
||||
Those are the main ones:
|
||||
|
||||
* `Selrect` and other geometric attributes (x, y, width, height...) define the
|
||||
position in the diagram and the bounding box.
|
||||
* `Transform` is a [2D transformation matrix](https://www.alanzucconi.com/2016/02/10/tranfsormation-matrix/)
|
||||
to rotate or stretch the shape.
|
||||
* `Constraints` explains how the shape changes when the container shape resizes
|
||||
(kind of "responsive" behavior).
|
||||
* `Interactions` describe the interactive behavior when the shape is displayed
|
||||
in the viewer.
|
||||
* `Fill` contains the shape fill color and options.
|
||||
* `Stroke` contains the shape stroke color and options.
|
||||
* `Shadow` contains the shape shadow options.
|
||||
* `Blur` contains the shape blur options.
|
||||
* `Font` contains the font options for a shape of type text.
|
||||
* `Content` contains the text blocks for a shape of type text.
|
||||
* `Exports` are the defined export settings for the shape.
|
||||
|
||||
Also a shape contains a reference to its containing shape (parent) and of all
|
||||
the children.
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.4. Backend Guide
|
||||
title: 5.4. Backend Guide
|
||||
---
|
||||
|
||||
# Backend guide #
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.2. Common Guide
|
||||
title: 5.2. Common Guide
|
||||
---
|
||||
|
||||
# Common guide
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.5. Data Guide
|
||||
title: 5.5. Data Guide
|
||||
---
|
||||
|
||||
# Data Guide
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.1. Dev environment
|
||||
title: 5.1. Dev environment
|
||||
---
|
||||
|
||||
# Development environment
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.3. Frontend Guide
|
||||
title: 5.3. Frontend Guide
|
||||
---
|
||||
|
||||
# Frontend Guide
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4. Developer Guide
|
||||
title: 5. Developer Guide
|
||||
---
|
||||
|
||||
# Developer Guide
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.6.2 Assets storage
|
||||
title: 5.6.2 Assets storage
|
||||
---
|
||||
|
||||
# Assets storage
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.6.1 Authentication
|
||||
title: 5.6.1 Authentication
|
||||
---
|
||||
|
||||
# User authentication
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 4.6. Penpot subsystems
|
||||
title: 5.6. Penpot subsystems
|
||||
---
|
||||
|
||||
# Penpot subsystems
|
||||
|
||||
Loading…
Reference in New Issue
Block a user