penpotqa

QA Test for Penpot

Based on Playwright framework.

1. Initial requirements and configuration.

Prerequisites for local run:

• Windows OS
• Screen resolution 1920x1080
• Installed Node.js
• “Clear” Penpot account (without added files, projects, etc., but with a completed onboarding flow).
• The .env file added to the root of the project with 3 env variables:
  • LOGIN_EMAIL (email from your Penpot account)
  • LOGIN_PWD (password from your Penpot account)
  • BASE_URL (Penpot url - e.g. http://localhost:9001/ if deployed locally)

2. Test run - main notes.

Upon cloning the repo and trying to run tests, you may be prompted to install the browsers: npx playwright install By default, npm test runs all tests in Chrome browser (the script "test": "npx playwright test --project=chrome" in package.js). To run the specific test/tests, change the test script in package.js in the next ways (or add a separate script):

• Run single test (by title) - e.g. "npx playwright test -g \"CO-154 Transform ellipse to path\" --project=chrome"
• Run single test spec (file) - e.g. "npx playwright test tests/login.spec.js --project=chrome"
• Run specific tests package (folder) - e.g. "npx playwright test tests/dashboard --project=chrome"

To run the tests in Firefox and Webkit browsers, use "firefox" and "webkit" scripts accordingly: "firefox": "npx playwright test --project=firefox" "webkit": "npx playwright test --project=webkit"

Currently, there are 318 tests at all. For each browser there is the following execution time:

• Chrome - 75 min
• Firefox - 80 min
• Webkit - 90 min

3. Test run - additional settings.

Some settings from playwright.config.js may be useful:

• By default, test retries are enabled (test retries 2 times in case of failure). To disable them, change the value of retries property to 0
• timeout and expect.timeout - maximum time of execution a single test and of a waiting expect() condition to be met accordingly
• use.headless - change to false to run in headed browser mode
• use.channel: "chrome" - comment out to run tests in Chromium instead of Chrome (for "chrome" project)
• workers: 1 - for now tests are running sequentially (parallelism is disabled)

4. Snapshots comparison.

Expected snapshots are stored in tests/{spec-name}-snapshots/{project-name} folders (where project-name is the browser name). In most of the cases, they capture and compare not the whole visible area of the screen but only the single element/section (e.g. created shape or canvas with created board). It helps to avoid the failure of the tests upon such changes in the UI like adding new sections to the Design panel, new buttons to the toolbars and so on. Such tests use the pattern: await expect(<pageName.elementName>).toHaveScreenshot("snapshotName.png"); However, about 10% of the tests capture and compare all visible area of the screen, since in such scenarios it makes sense to check not only the layers/canvas, but the panels, toolbar, etc. These tests are use the pattern: await expect(page).toHaveScreenshot("snapshotName.png", { mask: [pageName.elementName], }); Masking is used in order to ignore the elements which have unpredictable text content or values (e.g. username, timestamp, etc.). Therefore, however the impact of future UI changes to snapshots comparison is minimized, it is impossible to avoid such cases at all. However, it is rather simple to update snapshots:

• Upon test failure, compare the actual and expected snapshots and verify that the difference occurred due to intended changes in UI.
• Delete expected snapshot from folder.
• Run the test one more time, which will capture the actual snapshot and write it as expected to the appropriate folder.
• Commit the new expected snapshot and push.

Note 1: there is a known issue that Chrome does render differently in headless and headed modes, that's why expect.toHaveScreenshot.maxDiffPixelRatio: 0.01 is set in playwright.config.js for "chrome" project , which means that an acceptable ratio of pixels that are different to the total amount of pixels is 1% within screenshot comparison.

Note 2: expected screenshots for Firefox represent headless mode. Since scrollbars are hidden in headless mode and there is no way to unhide them in Firefox (unlike Chrome) - screenshots comparison in headed Firefox mode may fail in a few tests.

5. Performance testing.

To exclude performance tests from the periodical regression test run the following scripts should be used:

• for Chrome: "npx playwright test --project=chrome -gv 'PERF'"
• for Firefox: "npx playwright test --project=firefox -gv 'PERF'"
• for Webkit: "npx playwright test --project=webkit -gv 'PERF'"

Note: The above scripts should be executed via the command line. Do not run them directly from the package.json, because in such way performance tests are not ignored.

6. Running tests via GitHub Actions.

On Settings > Environments page 2 environments were created: PRE and PRO. For each environment the appropriate secrets were added:

• LOGIN_EMAIL (email from your Penpot account, which is used for tests)
• LOGIN_PWD (password from your Penpot account, which is used for tests)
• BASE_URL (Penpot url)

2 .yml files were added into .github/workflows directory with settings for environments:

• tests for PRE env will be run by schedule: each Thursday at 6:00 am UTC (and also it is possible to trigger them manually)
• tests for PRO env will be run only by request and triggered manually

Note:

• The UTC time is used for schedule set up.
• There may be a delay for start running tests by schedule. It will take nearly 5-15 minutes.

There are 2 workflows on Actions tab:

• Penpot Regression Tests on PRO env
• Penpot Regression Tests on PRE env

To run workflow by request you need to open it from the left sidebar and click on [Run workflow] > [Run workflow]. In a few seconds running should be start.

Tests run results:

When the run will be finished the appropriate marker will appear near the current workflow:

• green icon - workflow has been passed
• red icon - workflow has been failed

It is possible to open workflows (both passed and failed) and look through the Summary info:

• Status
• Total duration
• Artifacts

In Artifacts section there will be a 'playwright-report.zip' file. It is possible to download it, extract and open index.html file with the default playwright report.