penpotqa/README.md

# penpotqa

QA Test for Penpot

Based on Playwright framework.

**1. Initial requirements and configuration.**

Prerequisites for local run:

- Windows OS
- Screen resolution 1920x1080
- Installed Node.js
- “Clear” Penpot account (without added files, projects, etc., but with a completed onboarding flow).
- The _.env_ file added to the root of the project with 3 env variables:
  - `LOGIN_EMAIL` (email from your Penpot account)
  - `LOGIN_PWD` (password from your Penpot account)
  - `BASE_URL` (Penpot url - e.g. http://localhost:9001/ if deployed locally)

**2. Test run - main notes.**

Upon cloning the repo and trying to run tests, you may be prompted to install the browsers:
`npx playwright install`
By default, `npm test` runs all tests in Chrome browser (the script `"test": "npx playwright test --project=chrome"` in _package.js_).
To run the specific test/tests, change the test script in _package.js_ in the next ways (or add a separate script):

- Run single test (by title) - e.g. `"npx playwright test -g \"CO-154 Transform ellipse to path\" --project=chrome"`
- Run single test spec (file) - e.g. `"npx playwright test tests/login.spec.js --project=chrome"`
- Run specific tests package (folder) - e.g. `"npx playwright test tests/dashboard --project=chrome"`

To run the tests in Firefox and Webkit browsers, use `"firefox"` and `"webkit"` scripts accordingly:
`"firefox": "npx playwright test --project=firefox"`
`"webkit": "npx playwright test --project=webkit"`

Currently, there are 256 tests for regression suite, average time of execution per one browser - 60 minutes.

**3. Test run - additional settings.**

Some settings from _playwright.config.js_ may be useful:

- By default, test retries are enabled (test retries 2 times in case of failure). To disable them, change the value of `retries` property to 0
- `timeout` and `expect.timeout` - maximum time of execution a single test and of a waiting expect() condition to be met accordingly
- `use.headless `- change to _false_ to run in headed browser mode
- `use.channel: "chrome"` - comment out to run tests in Chromium instead of Chrome (for "chrome" project)
- `workers: 1` - for now tests are running sequentially (parallelism is disabled)

**4. Snapshots comparison.**

Expected snapshots are stored in _tests/{spec-name}-snapshots/{project-name}_ folders (where project-name is the browser name).
In most of the cases, they capture and compare not the whole visible area of the screen but only the single element/section (e.g. created shape or canvas with created board).
It helps to avoid the failure of the tests upon such changes in the UI like adding new sections to the Design panel, new buttons to the toolbars and so on.
Such tests use the pattern:
`await expect(<pageName.elementName>).toHaveScreenshot("snapshotName.png");`
However, about 10% of the tests capture and compare all visible area of the screen, since in such scenarios it makes sense to check not only the layers/canvas, but the panels, toolbar, etc.
These tests are use the pattern:
`await expect(page).toHaveScreenshot("snapshotName.png", { mask: [pageName.elementName], });`
Masking is used in order to ignore the elements which have unpredictable text content or values (e.g. username, timestamp, etc.).
Therefore, however the impact of future UI changes to snapshots comparison is minimized, it is impossible to avoid such cases at all.
However, it is rather simple to update snapshots:

- Upon test failure, compare the actual and expected snapshots and verify that the difference occurred due to intended changes in UI.
- Delete expected snapshot from folder.
- Run the test one more time, which will capture the actual snapshot and write it as expected to the appropriate folder.
- Commit the new expected snapshot and push.

Note 1: there is a known issue that Chrome does render differently in headless and headed modes, that's why
`expect.toHaveScreenshot.maxDiffPixelRatio: 0.01` is set in _playwright.config.js_ for "chrome" project , which means that
an acceptable ratio of pixels that are different to the total amount of pixels is 1% within screenshot comparison.

Note 2: expected screenshots for Firefox represent headless mode. Since scrollbars are hidden in headless mode and
there is no way to unhide them in Firefox (unlike Chrome) - screenshots comparison in headed Firefox mode may fail in a few tests.

**4. Performance testing.**

To exclude performance tests from the periodical regression test run the following scripts should be used:
- for Chrome: `"test": "npx playwright test --project=chrome -gv 'PERF'"`
- for Firefox: `"firefox": "npx playwright test --project=firefox -gv 'PERF'"`
- for Webkit: `"webkit": "npx playwright test --project=webkit -gv 'PERF'"`

**5. Running tests via GitHub Actions.**

On _Settings > Environments_ page 2 environments were created: _PRE_ and _PRO_.
For each environment the appropriate secrets were added:
  - _LOGIN_EMAIL_ (email from your Penpot account, which is used for tests)
  - _LOGIN_PWD_ (password from your Penpot account, which is used for tests)
  - _BASE_URL_ (Penpot url)

2 _.yml_ files were added into _.github/workflows_ directory with settings for environments:
  - tests for _PRE_ env will be run by schedule: each Friday at 8:00 am UTC (and also it is possible to trigger them manually)
  - tests for _PRO_ env will be run only by request and triggered manually

**Note**:
- The UTC time is used for schedule set up.
- There may be a delay for start running tests by schedule. It will take nearly 5-15 minutes.

There are 2 workflows on _Actions_ tab:
- Penpot Regression Tests on PRO env
- Penpot Regression Tests on PRE env

To run workflow by request you need to open it from the left sidebar and click on _[Run workflow]_ > _[Run workflow]_.
In a few seconds running should be start.

**Tests run results:**

When the run will be finished the appropriate marker will appear near the current workflow:
- `green icon` - workflow has been passed
- `red icon` - workflow has been failed

It is possible to open workflows (both passed and failed) and look through the _Summary_ info:
- Status
- Total duration
- Artifacts

In _Artifacts_ section there will be a _'playwright-report.zip'_ file. It is possible to download it, extract and open _index.html_ file with the default playwright report.