diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 74544083..fc65f698 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -9,14 +9,14 @@ assignees: '' **Please remove this line and all the following lines when submit issue** -In order to solve your issue quickly, please read this page and try common solutions in the section `Asking questions in GitHub` -https://pyarmor.readthedocs.io/en/latest/questions.html +In order to solve your issue quickly, please refer to `Ask Pyarmor` in this page +https://pyarmor.eke.org.cn/ -If it's really a bug, please following the guide in the section `Reporting issues` +If it's really a bug, please following the guide in the section `Report issue` -为了快速解决你遇到的问题，请务必打开下面的链接并尝试其中关于常见问题的通用解决方案 -https://pyarmor.readthedocs.io/en/latest/questions.html +为了快速解决你遇到的问题，请参考下面页面中的 `解疑答惑` +https://pyarmor.eke.org.cn/ -如果确认是一个 Bug，建议按照上面链接中的章节 `报告问题` 的规范提交问题报告 +如果确认是一个 Bug，请按照 `报告问题` 的规范提交问题报告 **在提交报告时候请删除当前行以及上面的所有行** diff --git a/.gitignore b/.gitignore index 83ac52fd..53fa5280 100644 --- a/.gitignore +++ b/.gitignore @@ -102,4 +102,4 @@ ENV/ # pip .pypirc -pypi-upload.sh +scripts/pypi-upload.sh diff --git a/.readthedocs.yaml b/.readthedocs.yaml index f61cd63b..043e2783 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -5,6 +5,9 @@ build: tools: python: "3.7" +sphinx: + configuration: docs/conf.py + python: install: - requirements: docs/requirements.txt diff --git a/LICENSE b/LICENSE index 44eac87e..57ccfbeb 100755 --- a/LICENSE +++ b/LICENSE @@ -10,7 +10,7 @@ accessing or in any other way using Pyarmor - referred to as "user" or 1.1. "This Software" - means Pyarmor 8.0, it doesn't include Pyarmor prior to 8.0. Pyarmor 8.0 is + means Pyarmor 8.0+, it doesn't include Pyarmor prior to 8.0. Pyarmor 8.0 is rewritten, and provides new features. For compatibility, Pyarmor 8.0 also includes most of the old features, this license doesn't apply to those old features. It's only for new features. @@ -71,6 +71,9 @@ accessing or in any other way using Pyarmor - referred to as "user" or 1.16 "Licensed Product" means product has been registered to one Pyarmor License. +1.17 "CI License" + means a kind of Pyarmor License + 2. License Grants and Conditions -------------------------------- @@ -88,8 +91,8 @@ but there are some limitations. 2.1.2 Can not use feature Mix Str 2.1.3 Can not use feature RFT Mode, BCC Mode 2.1.4 Can not be used to obfuscate any commercial product. If the total sale - income of this product is less than 30x license fees, this software could - be used temporarily. + income of this product is less than 100 x license fees, this software + could be used temporarily. 2.1.5 Can not be used to provide obfuscation service in any form, in short this software can't be used to obfuscate the scripts of others @@ -97,18 +100,27 @@ but there are some limitations. User purchases Pyarmor License to unlock these limitations except 2.1.5. -Licensor issues 3 kind of Pyarmor License. +Licensor issues 4 kind of Pyarmor License. 2.2.1 Pyarmor Basic, unlock limitations 2.1.1, 2.1.2, 2.1.4 2.2.2 Pyarmor Pro, unlock limitations 2.1.1, 2.1.2, 2.1.3, 2.1.4 2.2.3 Pyarmor Group, unlock limitations 2.1.1, 2.1.2, 2.1.3, 2.1.4 +2.2.4 Pyrmor CI, unlock limitations 2.1.1, 2.1.2, 2.1.3, 2.1.4 Pyarmor Basic and Pyarmor Pro need internet connection to verify, it doesn't work in the device without internet connection. +Pyarmor Pro can not be used in CI/CD pipeline. + Pyarmor Group need not internet connection. -Each Pyarmor License only need pay once, not periodically. +Pyarmor CI can only be used in CI/CD pipeline or docker container. + +Pyarmor Basic, Pro and Group only need pay once, not periodically. + +Pyarmor CI will be expired after one year since purchased date, +licensor won't charge user automatically, but send new invoice to user +before the expiration date to get paid. 2.3 Conditions @@ -116,14 +128,13 @@ Each Pyarmor License only need pay once, not periodically. Licensed Product has an unique License No. in global. If user has many products, each product need purchase one Pyarmor License. Except 2.6.1 -2.3.2 Pyarmor License could be installed in many machines and devices which - used to develop, build, debug, test and support Licensed Product. But - there is limitation to be used at the same time. See 2.3.3 +2.3.2 Pyarmor License could be installed in limited machines and devices which + used to develop, build, debug, test and support Licensed Product. 2.3.3 Pyarmor Basic and Pro License can only be used in no more than 100 - devices in 24 hours. Pyarmor License be used means use any feature of - Pyarmor in one machine. Running obfuscated scripts generated by - Pyarmor is not considered as Pyarmor License be used. + devices. Pyarmor License be used means use any feature of Pyarmor in + one machine. Running obfuscated scripts generated by Pyarmor is not + considered as Pyarmor License be used. 2.3.4 Pyarmor Basic and Pro License need internet connection to verify. @@ -162,9 +173,9 @@ copyright doctrines of fair use, fair dealing, or other equivalents. 2.6.1 If user has many products, and has purchased one license for the first product. The second product could use first product license only if sale - income of the second product less than 30x license fees. Once greater than - 30x license fees, the second product need purchase its own license. It's - same to user's other products. + income of the second product less than 100x license fees. Once greater + than 100x license fees, the second product need purchase its own license. + It's same to user's other products. 3. Responsibilities and Commitments ----------------------------------- diff --git a/LICENSE-ZH b/LICENSE-ZH index 18396d2d..92bc6bb4 100755 --- a/LICENSE-ZH +++ b/LICENSE-ZH @@ -105,8 +105,8 @@ BCC 加密模式 (2) 混淆字符串功能在试用版中无法使用。 (3) RFT 加密模式，BCC 加密模式在试用版无法使用。 (4) 不可以应用于加密商用产品。特别的，如果商用产品的累计销售额小于基础版许 - 可证费用乘以30，可以暂时使用；但是一旦累计销售额超过阀值，就不可以在继 - 续使用。 + 可证费用乘以 100，可以暂时使用；但是一旦累计销售额超过阀值，就不可以在 + 继续使用。 (5) 运行辅助包的名称 "pyarmor_runtime_000000" 不可以被设置和修改 (6) 不可以使用本软件提供任何形式的加密服务，不管是通过应用程序还是网络服务。 总之在任何情况下都不允许使用本软件加密其他人的脚本。 @@ -122,7 +122,7 @@ BCC 加密模式 是说，任何一种使用本软件进行保护的产品都有自己唯一的许可证编号，不允许两种不同产 品使用相同的许可证编号。 -本软件提供三种许可证，分别解锁不同的功能 +本软件提供四种许可证，分别解锁不同的功能 2.2.1 基础版许可证 @@ -136,12 +136,20 @@ BCC 加密模式 专家版许可证加密脚本的时候需要在线验证许可证 +专家版许可证不可用于 CI/CD 管线或者 Docker 容器中 + 2.2.3 集团版许可证 集团版许可证解锁限制 2.1 中的 (1) (2) (3) (4) (5) (7). 集团版许可证加密脚本的时候不需要在线验证许可证 +2.2.4 管道版许可证 + +管道版许可证解锁限制 2.1 中的 (1) (2) (3) (4) (5) (7). + +管道版许可证仅用于 CI/CD 管线或者 Docker 容器中 + 不管哪一种许可证，运行加密脚本的时候都无需验证许可证，本软件对于加密脚本的运行没 有任何控制和限制。 @@ -150,8 +158,12 @@ BCC 加密模式 除了购买软件许可的费用之外，没有其它任何费用。获得许可的用户可以使用本软件在许可 的范围之内加密任何脚本并自由发布，不需要在向许可人支付任何费用。 -购买软件许可的费用是一次性收费，可以永久在购买本软件时候的版本中使用，但是许可证 -可能在任何一个升级版本中失效，许可人不承诺许可证可以在今后所有的升级版本中使用。 +除了管线版许可证，购买软件许可的费用是一次性收费，可以永久在购买本软件时候的版本 +中使用，但是许可证可能在任何一个升级版本中失效，许可人不承诺许可证可以在今后所有 +的升级版本中使用。 + +对于管线版许可证，自购买之日起一年内有效，可以使用在有效期之内发布的任何版本。在 +许可证到期之前，许可人会发送邮件通知用户续费。 一旦激活许可证之后，不在支持退款，购买之后没有激活许可证，在三十天之内支持取消许 可证并退款。但是如果购买时间超过三十天，没有激活的许可证也不再支持退款。 @@ -161,11 +173,11 @@ BCC 加密模式 如果用户有多个产品并且已经为第一个产品购买许可证，其他产品满足下列条件可以使用第 一个产品的许可证： -(1) 如果该产品的销售收入在当前许可证费用的10倍之内，那么该产品可以使用第一个产品 - 的许可证。 +(1) 如果该产品的销售收入在当前许可证费用的 100 倍之内，那么该产品可以使用第一个 + 产品的许可证。 -(2) 如果该产品的销售输入超过当前许可证费用的10倍，那么该产品不可以继续使用第一个 - 产品的许可证，需要单独购买新的许可证。 +(2) 如果该产品的销售输入超过当前许可证费用的 100 倍，那么该产品不可以继续使用 + 第一个产品的许可证，需要单独购买新的许可证。 本许可证并不旨在限制用户根据适用条款享有的任何权利、公平交易或其他等效的合理使用。 @@ -200,9 +212,8 @@ BCC 加密模式 4.2 用户可以修改本软件相关的 Python 脚本以满足使用方面的额外需求，但是修改后的脚 本只能在许可证范围内使用，不得分发给其它人 -4.3 同一个许可下面，同时使用本软件的设备数目不超过 100 个。 同时使用本软件是指从 - 现在开始在 24 小时内曾经运行 pyarmor 命令的设备。这里的设备是指安装本软件并 - 使用本软件对脚本进行加密的设备，不是指运行加密脚本的客户机器。 +4.3 同一个许可下面，使用本软件的设备数目有限制。这里的设备是指安装本软件并使用本 + 软件对脚本进行加密的设备，不是指运行加密脚本的客户机器。 4.4 每一个许可证绑定到一种产品，不可以转移给其它产品。 diff --git a/README.md b/README.md index 65783aba..0efd9082 100644 --- a/README.md +++ b/README.md @@ -66,92 +66,56 @@ Please read the [Pyarmor EULA](LICENSE). ## Getting Help -1. **Consult the [Pyarmor 8.0 Documentation][doc].** -2. **Check the [FAQ][faq] for answers to common questions.** -3. **Try the documentation [index][genindex] or the [detailed table of contents][mastertoc].** -4. **If you still can't find the information you need, see [asking questions on GitHub][asking].** -5. **[Report bugs][issues] following the issue template.** -6. **For business and security inquiries, send an email to .** +- **[Ask in learning system][learning] or [look through check list][checklist]** +- **Consult the [Pyarmor Documentation][doc].** +- **Check the [FAQ][faq] for answers to common questions.** +- **Try the documentation [index][genindex] or the [detailed table of contents][mastertoc].** +- **If you still can't find the information you need, see [asking questions on GitHub][asking].** +- **[Report bugs][issues] following the issue template.** +- **For business and security inquiries, send an email to .** -[faq]: https://pyarmor.readthedocs.io/en/latest/questions.html -[issues]: https://github.com/dashingsoft/pyarmor/issues -[genindex]: https://pyarmor.readthedocs.io/en/stable/genindex.html -[mastertoc]: https://pyarmor.readthedocs.io/en/stable/index.html#table-of-contents -[asking]: https://pyarmor.readthedocs.io/en/latest/questions.html#asking-questions-in-github -[doc]: https://pyarmor.readthedocs.io/ +There is also one third-party learn platform + +- **[Ask Pyarmor Guru][gurubase], it is a Pyarmor-focused AI to answer your questions** (not made by Pyarmor Team, the answer doesn't stand for Pyarmor Team's opinion) ## Resources * [Website](https://pyarmor.dashingsoft.com) -* [Documentation 8.0][doc] +* [Documentation][doc] +* [Documentation 8.x](https://pyarmor.readthedocs.io/en/v8.5.12/) * [Documentation 7.x](https://pyarmor.readthedocs.io/en/v7.7/) +* [Pyarmor 9.1 new features](https://pyarmor.eke.org.cn/v9.html) +* [Pyarmor Learning System][learning] 中文资源 * [Pyarmor 网站](https://pyarmor.dashingsoft.com/index-zh.html) -* [Pyarmor 8.0 在线文档](https://pyarmor.readthedocs.io/zh/latest/) +* [Pyarmor 在线文档](https://pyarmor.readthedocs.io/zh/latest/) +* [Pyarmor 8.x 在线文档](https://pyarmor.readthedocs.io/zh/v8.5.12/) * [Pyarmor 7.x 在线文档](https://pyarmor.readthedocs.io/zh/v7.x/) +* [Pyarmor 9.1 新功能](https://pyarmor.eke.org.cn/v9.html) +* [Pyarmor 学习系统][learning] ## Changelog -Pyarmor 8.0 introduces significant changes. It has been rewritten and new features are implemented through the new commands: -`gen`, `reg`, `cfg`. These commands only work for Python 3.7 and above. - -Users of versions prior to 8.0 should read the [Import Notes][important-notes] section to decide whether to upgrade Pyarmor. - Each major version comes with a separate changelog file, detailing fixed issues, new features, and compatibility issues between different versions. Make sure to read the changelog carefully before upgrading Pyarmor: + - [Pyarmor 8.x Changelog](docs/ChangeLogs.8) +- [Pyarmor 9.x Changelog](docs/ChangeLogs.9) **Full changelogs** at [releases][releases] -**Upcoming features** at [Pyarmor 8.x Release Plan](ReleasePlan.md) +**Upcoming features** at [Pyarmor Release Plan](docs/ReleasePlan.md) [releases]: https://github.com/dashingsoft/pyarmor/releases - -[important-notes]: #important-notes-for-users-of-pyarmor-prior-to-80 - -## Important Notes for Users of Pyarmor Prior to 8.0 - -Going forward, only bug fixes will be provided for older commands, such as `obfuscate` and `licenses`. -No new features will be added to these commands, but they will continue to be usable. - -Upon the release of Pyarmor 8.0, there are three scenarios for existing users: - -### 1. Never upgrade to version 8.0+ - -- **SPP mode will not work** - - To use SPP mode, you must upgrade Pyarmor to version 8.0 or later. - -- **`pyarmor register` command without arguments returns a `404` error** - - This command was used to query registration information in earlier versions of Pyarmor. However, the license server no longer supports this web API. Use `pyarmor -v` to ensure you are not using a trial version. - -- **Registering Pyarmor with `pyarmor register pyarmor-regcode-xxxxxx.txt` is limited to 10 uses**: - - To use Pyarmor on a new machine, CI server, or Docker, refer to the second method described in the registration file "pyarmor-regcode-xxxxxx.txt": -``` -Downloading "pyarmor-regfile-xxxxxx.zip" once, use this `.zip` file to register Pyarmor later. -``` - -### 2. Upgrade to version 8.0 but only use old features - -By default, the `pyarmor` command only accepts the new commands. - -To continue using older commands like `obfuscate` and `licenses`, you can: -- Use `pyarmor-7` instead of `pyarmor` -- Set the environment variable `PYARMOR_CLI=7` and continue using `pyarmor` -- Call the entry point `pyarmor.pyarmor:main_entry` in any other way - -### 3. Upgrade to version 8.0 and use new features - -- **New EULA**: - - Adhere to the new [Pyarmor EULA](LICENSE). This is a significant change for users with old personal licenses, as the new license only allows one product. - -- **License Upgrades**: - - Not all old licenses can be freely upgraded to the new license. Please refer to the [Pyarmor licenses][licenses] for more information. - -- **Internet Connection**: - - Older commands do not require an internet connection, but new commands do. - -- **Python Version Support**: - - While older commands support Python 2.7-3.10, new commands only support Python 3.7+. +[faq]: https://pyarmor.readthedocs.io/en/latest/questions.html +[issues]: https://github.com/dashingsoft/pyarmor/issues +[genindex]: https://pyarmor.readthedocs.io/en/stable/genindex.html +[mastertoc]: https://pyarmor.readthedocs.io/en/stable/index.html#table-of-contents +[asking]: https://pyarmor.readthedocs.io/en/latest/questions.html#asking-questions-in-github +[doc]: https://pyarmor.readthedocs.io/ +[gurubase]: https://gurubase.io/g/pyarmor +[learning]: https://pyarmor.eke.org.cn/ +[checklist]: https://pyarmor.readthedocs.io/en/latest/reference/solutions.html diff --git a/docs/ChangeLogs.8 b/docs/ChangeLogs.8 index 04eddd49..59351361 100755 --- a/docs/ChangeLogs.8 +++ b/docs/ChangeLogs.8 @@ -3,6 +3,9 @@ In order to improve security and support Python 3.11, there are significant changes in Pyarmor 8.0 +It has been rewritten and new features are implemented through the new commands: +`gen`, `reg`, `cfg`. These commands only work for Python 3.7 and above. + ## New features * Support Python 3.11 @@ -68,4 +71,7 @@ There are 3 cases for old users after Pyarmor 8.0 is released: Python 3.7+ Refer to -https://github.com/dashingsoft/pyarmor#import-notes-for-pyarmor-prior-to-80 +https://github.com/dashingsoft/pyarmor/tree/v8.0#import-notes-for-pyarmor-prior-to-80 +https://github.com/dashingsoft/pyarmor/releases/tag/v8.0 + +[licenses]: https://pyarmor.readthedocs.io/en/latest/licenses.html diff --git a/docs/ChangeLogs.9 b/docs/ChangeLogs.9 new file mode 100755 index 00000000..d0a7d557 --- /dev/null +++ b/docs/ChangeLogs.9 @@ -0,0 +1,79 @@ +# Pyarmor 9 Change Logs + +Because Pyarmor License is paid once in Pyarmor 8, but online verification for using Pyarmor License in CI/CD pipeline will continue consuming a lot of Pyarmor Server resources. + +So Pyarmor 9 has a big change about using Pyarmor in CI/CD pipeline: + +- Pyarmor Pro License couldn't be used in CI/CD pipeline since Pyarmor 9 +- Pyarmor CI License is right for CI/CD pipeline case +- Pyarmor Basic still could be used in CI/CD pipeline, but it need request one extra CI regfile + +In a short + +**If not use Pyarmor License in CI/CD pipeline, almost nothing changed** +**If not upgrade to Pyarmor 9, but still use Pyarmor 8, nothing changed** +**Pyarmor Pro License purchased after 2024-10-20 could not be used in CI/CD pipeline even use it with Pyarmor 8** +**Pyarmor CI License is paid each year, and couldn't be used in local machine** + +## Upgrade Basic or Pro License to Pyarmor 9 + +1. If Pyarmor License has been registered in this device + + - First upgrade to Pyarmor 9 + ```bash + $ pip install -U pyarmor + ``` + + - When first time to obfuscate scripts, it will show hints + ```bash + $ pyarmor gen foo.py + + ... + Pyarmor 9 has big change on CI/CD pipeline + If not using Pyarmor License in CI/CD pipeline + Press "c" to continue + Otherwise press "h" to check Pyarmor 9.0 Upgrade Notes + + Continue (c), Help (h), Quit (q): + ``` + - Just press `c` to continue, there is no prompt later + +2. If Pyarmor License isn't registered in this device + + - First use activation file to generate new registration file + ```bash + $ pip install -U pyarmor + + # Please replace XXX with real product name + $ pyarmor reg -p XXX pyarmor-regcode-xxxx.txt + ``` + - Save and backup new registration file `pyarmor-regfile-xxxx.zip` + + - Use this new regfile to register Pyarmor in other new device + ```bash + $ pyarmor reg pyarmor-regfile-xxxx.zip + $ pyarmor -v + ``` + + **If activation file is used too many times, please first install Pyarmor 8, then upgrade to Pyarmor 9** + +## Upgrade Group License to Pyarmor 9 + +It need generate device regfile again with Pyarmor 9.0+ + +- First upgrade to Pyarmor 9 + ```bash + $ pip install -U pyarmor + ``` + +- Then generate device regfile as before + + For example, generate device regfile `pyarmor-device-regfile-6000.1.zip` for device no. 1 + ```bash + $ pyarmor reg -g 1 /path/to/pyarmor-regfile-6000.zip + ``` + +- Finally, use new one to register Pyarmor in offline device + ```bash + $ pyarmor reg pyarmor-device-regfile-6000.1.zip + ``` diff --git a/ReleasePlan.md b/docs/ReleasePlan.md similarity index 52% rename from ReleasePlan.md rename to docs/ReleasePlan.md index db58efea..87287260 100755 --- a/ReleasePlan.md +++ b/docs/ReleasePlan.md @@ -1,6 +1,11 @@ -# Release Plan for Pyarmor 8 +# Release Plan -The scheduled features for Pyarmor 8 +The scheduled features for Pyarmor 8+ + +- [Pyarmor 8 (closed)](#plan-features-for-pyarmor-8) +- [Pyarmor 9 (working)](#plan-features-for-pyarmor-9) + +## Plan Features For Pyarmor 8 **8.0** @@ -54,23 +59,62 @@ Status: released (2023-10-18) **8.5** -Status: first released at 2024-03-08, unstable, still developing +Status: released (2024-03-08) - Expert users could write their own python code in pyarmor_runtime initialization -- Pyarmor.man package with web-ui to help users learning, using Pyarmor, and quickly find solutions when something is wrong - New plugin "post_script" called after each script has been obfuscated - Runtime key could be obfuscated - No patch `pyarmor_runtime.so` for Apple Silcon so Windows users could build obfuscated scripts for Apple Silcon -**8.6** +## Plan Features For Pyarmor 9 + +**9.0** + +Status: released (2024-11-10) + +- Pyarmor.man package to find solutions quickly when something is wrong +- Add special Pyarmor CI License for CI/CD pipeline +- Support Python 3.13 (since v9.0.5) + +**9.1** + +Status: released (2025-02-24) + +- MINI mode, a simple RFT mode, the goal is high performance after obfuscation + +**9.2** + +Status: released (2025-10-31) + +- Support Python 3.14 and 3.15 (a2) + +**9.2.2** + +Status: released (2025-12-8) + +- ECC mode, mini BCC mode, only transform part of function body to C code +- VMC mode, simple ECC mode, convert function body to VM, not real C code + +The following features are cancelled because ECC/VMC mode are introduced: + +- Convert unsupported node types as far as possible for BCC mode (cancelld) +- Expert users could customize C sources generated by BCC mode (cancelld) +- BCC mode support cross-platform (may not if too complex) (cancelled) +- Obfuscate string constants in extension module "pyarmor_runtime" (cancelled) + +**9.3** + +Status: developing + +- Full testcases +- Refine RFT mode to improve performance for big scripts (same as MINI mode) +- `pyarmor build` supports option `--pack` to pack project to one executable +- RFT mode supports module name refactor + +**9.4** Status: not start -- Full test -- Refine RFT mode to improve performance for big scripts -- Convert current unsupported node types as far as possible for BCC mode -- Expert users could customize C functions generated by BCC mode -- BCC mode support cross-platform (may not if too complex) -- Obfuscate string constants in extension module "pyarmor_runtime" (draft) +- Learning system to help users learning and using Pyarmor -Pyarmor status will be stable by the end of 2024 (Dec. 31, 2024) +Pyarmor features will be stable in 2025 diff --git a/docs/_common_definitions.txt b/docs/_common_definitions.txt index db69a6b0..3443dd4b 100644 --- a/docs/_common_definitions.txt +++ b/docs/_common_definitions.txt @@ -4,6 +4,10 @@ .. _Pyarmor Doc: https://pyarmor.readthedocs.io/en/stable/ .. _Pyarmor 7.x Doc: https://pyarmor.readthedocs.io/en/v7.7/ .. _Pyarmor EULA: https://github.com/dashingsoft/pyarmor/blob/master/LICENSE +.. _MyCommerce Shopping: https://order.mycommerce.com/product?vendorid=200089125&productid=301044051 +.. _Pyarmor Shopping Cart: https://jondy.github.io/paypal/index.html +.. _Pyarmor V9 Man: https://pyarmor.eke.org.cn/archive/v9/docs/en/user/man.html +.. _Pyarmor V9 Doc: https://pyarmor.eke.org.cn/v9.html .. |pyarmor| replace:: :doc:`pyarmor ` .. |Pyarmor| replace:: :term:`Pyarmor` diff --git a/docs/conf.py b/docs/conf.py index 4fd2be47..74ae136b 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -49,7 +49,7 @@ # General information about the project. project = 'Pyarmor' -copyright = '2018 - 2024 Dashingsoft Corp.' +copyright = '2018 - 2025 Dashingsoft Corp.' author = 'Jondy Zhao' # The version info for the project you're documenting, acts as replacement for @@ -57,9 +57,9 @@ # built documents. # # The short X.Y version. -version = '8.5' +version = '9.2' # The full version, including alpha/beta/rc tags. -release = '8.5.12' +release = '9.2.7' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/how-to/ci.rst b/docs/how-to/ci.rst index e428d1c3..5dc4fa45 100644 --- a/docs/how-to/ci.rst +++ b/docs/how-to/ci.rst @@ -4,22 +4,178 @@ Using Pyarmor in CI Pipeline ============================== -Pyarmor could be used in CI/CD pipeline directly, but there are some limitions: +There are 2 ways to use Pyarmor in CI/CD pipeline: -- Group License generally doesn't work in CI/CD pipeline -- It only allows 3 runs in 1 minutes for Basic/Pro license -- It only allows 100 runs in 24 hours for Basic/Pro license +- Direct way, it's simple, but only works for Trial, Basic and CI license, and there is rate limits +- Indirect way, it need change the original build workflow, but works for any type license - - If need more than 100 runs, refer to :ref:`Using Pyarmor CI Quota` +.. note:: -Pyarmor recommends to use Pyarmor in CI/CD pipeline by this way: + In this page, CI/CD pipeline means the runner is unstable device like docker containter, which machine id is changed after reboot. -- First obfuscate the scripts by a few runner and store them to another branch like `master-obf` -- Then all the other runners continue the rest pipeline based on this branch like before + If the runner in CI/CD pipeline is physical machine, it's not in CI/CD pipeline defined by Pyarmor, it can be taken as one normal device. -Because only first step runners need register Pyarmor, so it could solve run limitions in most of cases. +Direct Way +========== -Suppose test-project locates at `https://github.com/dashingsoft/test-project`, the directory tree as follows:: +**Trial Version** could be used in CI/CD pipeline by one step:: + + pip install pyarmor + +For :term:`Pyarmor Basic` and :term:`Pyarmor CI` License + +- Refer to :ref:`initial registration`, first got :term:`registration file` like ``pyarmor-regfile-xxxx.zip`` +- In local device run the following command to request one CI regfile ``pyarmor-ci-xxxx.zip``:: + + $ pyarmor reg -C pyarmor-regfile-xxxx.zip + + Check CI license info in local machine:: + + $ pyarmor --home temp reg pyarmor-ci-xxxx.zip + +- In CI/CD pipeline, add 2 steps to register Pyarmor by CI regfile:: + + # Please replace "9.X.Y" with current Pyarmor version + pip install pyarmor=9.X.Y + pyarmor reg pyarmor-ci-xxxx.zip + + Check registration information in CI/CD pipeline:: + + pyarmor -v + +Notes + +* Do not request CI regfile in CI/CD pipeline +* CI regfile ``pyarmor-ci-xxxx.zip`` will be expired about in 360 days +* CI regfile may not work in future Pyarmor version +* Once CI regfile doesn't work, require new one +* One license can request <= 100 CI regfiles + +.. note:: + + In GitHub Action, it need one extra step, otherwise `CI license only works in CI/CD pipeline` + + 1. For Ubuntu, add this step:: + + - run: sudo mv /dev/disk /dev/disk-none + + 2. For Darwin, add this step:: + + - run: sudo mv /dev/rdisk0 /dev/rdisk0-none + + Refer to this thread `Error when using CI license in CI pipeline `_ + +.. important:: + + In CI/CD pipeline, each run `pyarmor gen` will send license and docker information to Pyarmor License Server, excessive requests or requests beyond normal usage may be rejected by Pyarmor License Server. Generally do not exceed any of these rate limits: + + - 1 run per second + - 100 runs per hour + - 1,000 runs per day + - 10,000 runs per month + + If exceeds any of these limitions, please check the section `High frequency use solution` + +.. important:: + + It's not allowed to install and register Pyarmor in your customer's docker image, Pyarmor CI license is only be used in the build device. + +When need to request new CI regfile +----------------------------------- + +In the following cases, it need request one new CI regfile + +- After :term:`Pyarmor CI` License is expired, all the previous CI regfiles don't work any longer. After the renewal is successful, it need request new CI regfile ``pyarmor-ci-N.zip`` +- After Pyarmor is upgrade one new major/minor version, the old CI regfile may not work in the latest version (but it still works with old Pyarmor version). It need request one new CI regfile by new version. Note that the patch number has no effect for this case, for example, from v9.1.3 to v9.1.8, nothing changed. + +High frequency use solution +--------------------------- + +.. versionadded:: 9.2.0 + +If many `pyarmor gen` commands are used in one workflow, try to merge them to one + +For example:: + + # Old workflow: there are 3 "pyarmor gen" + pyarmor gen -R /path/to/package1 + pyarmor gen -R /path/to/package2 + pyarmor gen -R /path/to/package3 + + # New workflow: merge 3 to one + pyarmor gen -R /path/to/package1 /path/to/package2 /path/to/package2 + +Or create one Python script to execute all pyarmor commands in one process + +For example, create one script `batch.py`: + +.. code-block:: python + + import os + import shlex + + from pyarmor.cli.__main__ import main_entry as pyarmor_run + + # Do not run `pyarmor reg pyarmor-ci-XXXX.zip` in the script + + # Run command: pyarmor gen -R /path/to/package1 + pyarmor_run(['gen', '-R', '/path/to/package1']) + + # Or more like shell command to run: pyarmor gen -R /path/to/package2 + cmdlist = shlex.split("pyarmor gen -R /path/to/package2") + pyarmor_run(cmdlist[1:]) + + # Or change path + os.chdir('/path/to/other') + + # Execute any other pyarmor command + cmdlist = shlex.split("pyarmor gen key -e 30") + pyarmor_run(cmdlist[1:]) + +Then execute it in the workflow:: + + $ pyarmor reg pyarmor-ci-8000.zip + $ python3 batch.py + +If merge solution doesn't work, or you don't want change the original workflow, it need request more quota. + +Request More Quota +~~~~~~~~~~~~~~~~~~ + +The free quota is 10, 000 runs per month, exceed free quota, it need extra fees: + +- 100,000 per month, extra fees: $10 for one year +- 200,000 per month, extra fees: $20 for one year +- 300,000 per month, extra fees: $30 for one year +- ... + +The steps to request more quota + +1. Send request to Pyarmor Team (pyarmor@163.com) + + Please provide the project information, web link, etc. Pyarmor team only use these information for internal review. + + And the quota per month. For example, 100, 000 per month. + +2. Pyarmor Team send you PayPal invoice of extra fees. + + For example, if you request quota is 100, 000 per month, Pyarmor team will send you $10 invoice. One year later, when this CI license is expired and need renew, Pyarmor Team will send you $90 + $10 = $100 invoice + +3. After invoice is paid, Pyarmor Team will send you notification email + +4. You need request one new CI regfile with Pyarmor 9.2+ + + Only Pyarmor 9.2+ support this feature. + +Indirect Way +============ + +:term:`Pyarmor Pro` and :term:`Pyarmor Group` License can't be used in CI/CD pipeline directly, but this works + +- First obfuscate the scripts in local device and store them to another branch like `master-obf` +- Then in CI/CD pipeline to check this new branch + +Here is an example, suppose test-project locates at `https://github.com/dashingsoft/test-project`, the directory tree as follows:: $ tree test-project @@ -32,7 +188,7 @@ Suppose test-project locates at `https://github.com/dashingsoft/test-project`, t │ └── __init__.py └── __init__.py -The first runner will obfuscate the scripts and store them into another branch. Here it's an example bash script: +In local device the scripts are obfuscated and are stored into another branch: .. code-block:: bash @@ -60,74 +216,6 @@ The first runner will obfuscate the scripts and store them into another branch. # Push new branch to remote server $ git push -u origin master-obf -For all the other runners, they need not install Pyarmor, just checkout branch `master-obf`, and work as before. - -.. _using pyarmor ci quota: - -Using Pyarmor CI Quota -====================== - -Pyarmor CI Quota is one experimental solution for this issue: - -- Basic/Pro license only allow 100 runs in CI/CD pipeline in 24 hours - -Each ci quota allows 2,000 extra runs after 100 runs in 24 hours. - -.. list-table:: Table-1. Pyarmor CI Quota Price - :header-rows: 1 - - * - Extra Runs - - Net Price($) - - Remark - * - 2,000 - - 10 - - - -1. First check your license no. by `pyarmor -v`:: - - $ pyarmor -v - Pyarmor 8.5.11 (pro), 005068, btarmor - - License Type : pyarmor-pro - License No. : pyarmor-vax-005068 - License To : Tester - License Product : btarmor - ... - -2. Then purchasing Pyarmor CI Quota in MyCommerce website - - https://order.mycommerce.com/product?vendorid=200089125&productid=301123145 - -3. When placing order, fill "License to" with Pyarmor License No. For example, ``pyarmor-vax-005068`` - -4. Once payment is completed, this license will has one Pyarmor CI Quota - - If no more than 100 runs in 24 hours, it doesn't consume CI Quota. - -.. list-table:: Table-2. Pyarmor CI Quota Usage Example - :header-rows: 1 - - * - Date - - Runs in CI/CD pipeline - - Used Quota - - Left Quota - - Remark - * - 2024-05-01 - - - - - - 2000 - - Init quota - * - 2024-05-02 - - 80 - - 0 - - 2000 - - No consume quota because 80 < 100 - * - 2024-05-03 - - 120 - - 20 - - 1980 - - Consume quota 20 = 120 - 100 - -5. When ci quota is exhausted, it need purchase new Pyarmor CI Quota +In CI/CD pipeline, it need not install Pyarmor, just checkout branch `master-obf`, and work as before. .. include:: ../_common_definitions.txt diff --git a/docs/how-to/register.rst b/docs/how-to/register.rst index 3f66bed7..cadaa3f3 100644 --- a/docs/how-to/register.rst +++ b/docs/how-to/register.rst @@ -16,36 +16,23 @@ Prerequisite First of all -1. An :term:`activation file` of :term:`Pyarmor License` like :file:`pyarmor-regcode-xxxx.txt`, refer to :doc:`../licenses` to purchase right one -2. Pyarmor 8.2+ +1. One :term:`activation file` of :term:`Pyarmor License`, refer to :doc:`../licenses` to purchase right one +2. One device has installed Pyarmor 9.0+ 3. Internet connection -4. Product name bind to this license, for non-commercial use, product name is ``non-profits`` +4. Product name which bind to this license -**If any firewall turns on** - -In Windows ``pytransform.pyd`` will connect to ``pyarmor.dashingsoft.com`` port ``80`` to request token for online obfuscation, in other platforms it is ``pytransform3.so``. Refer to firewall documentation to allow it to connect ``pyarmor.dashingsoft.com:80``. - -Using Pyarmor Basic or Pro -========================== - -Basic use steps: - -1. Using :term:`activation file` to initial registration, set product name bind to this license -2. Once initial registration completed, a :term:`registration file` is generated -3. Using :term:`registration file` to register Pyarmor in other devices +.. _initial registration: Initial registration --------------------- - -Using :option:`-p` to specify product name for this license, for non-commercial use, set product name to ``non-profits``. +==================== -Assume this license is used to protect your product ``XXX``, initial registration by this command:: +Any license need this step to request :term:`registration file` from Pyarmor License Server by :term:`activation file` like :file:`pyarmor-regcode-xxxx.txt`:: $ pyarmor reg -p "XXX" pyarmor-regcode-xxxx.txt -Pyarmor will show registration information and ask for your confirmation. If everything is fine, type :kbd:`yes` and :kbd:`Enter` to continue. Any other input aborts registration. +Using :option:`-p` to specify product name for this license, please replace "XXX" with real product name. For non-commercial use, replace it to ``non-profits``. -If initial registration is successful, it prints final license information in the console. And a :term:`registration file` like :file:`pyarmor-regfile-xxxx.zip` is generated in the current path at the same time. This file is used for subsequent registration in other machines. +If initial registration is successful, one :term:`registration file` like :file:`pyarmor-regfile-xxxx.zip` is generated in the current path at the same time. This file is used for subsequent registration in other machines. Once initial registration completed, activation file :file:`pyarmor-regcode-xxxx.txt` is invalid, do not use it again. @@ -66,10 +53,13 @@ In 6 months real product name must be set by this command:: If it's not changed after 6 months, the product name will be set to ``non-profits`` automatically and can't be changed again. -Registering in other machines ------------------------------ +Using Pyarmor Basic or Pro +========================== -Copy :term:`registration file` :file:`pyarmor-regfile-xxxx.zip` to other machines, run the following command:: +1. Refer to :ref:`initial registration`, got :term:`registration file` like `pyarmor-regfile-xxxx.zip` +2. Using :term:`registration file` to register Pyarmor in other devices + +Copy :term:`registration file` to other machines, then run this command:: $ pyarmor reg pyarmor-regfile-xxxx.zip @@ -79,14 +69,84 @@ Check the registration information:: After successful registration, all obfuscations will automatically apply this license, and each obfuscation requires online license verification. -Registering in Docker or CI pipeline ------------------------------------- +This license can register Pyarmor on at most 100 devices + +On each device it's enough to register Pyarmor once, do not register Pyarmor before each obfuscation + +Do not register Pyarmor in the CI/CD pipeline or docker container by this :term:`registration file`, each run will taken as one new device. + +.. seealso:: :doc:`ci` + +.. _using ci license: + +Using CI License +================ + +.. versionadded:: 9.0 + +Refer to :ref:`initial registration`, got :term:`registration file` like `pyarmor-regfile-xxxx.zip` + +Do not use ``pyarmor-regfile-xxxx.zip`` in CI/CD pipeline directly, it's only used to request CI regfile: + +- In local device run the following command to request one CI regfile ``pyarmor-ci-xxxx.zip``:: + + $ pyarmor reg -C pyarmor-regfile-xxxx.zip + + Check CI license info in local machine:: + + $ pyarmor --home temp reg pyarmor-ci-xxxx.zip + +- In CI/CD pipeline, add 2 steps to register Pyarmor by CI regfile:: + + # Please replace "9.X.Y" with current Pyarmor version + pip install pyarmor=9.X.Y + pyarmor reg pyarmor-ci-xxxx.zip + + Check registration information in CI/CD pipeline:: + + pyarmor -v + +Notes + +* Do not request CI regfile in CI/CD pipeline +* CI regfile ``pyarmor-ci-xxxx.zip`` will be expired about in 360 days +* CI regfile may not work in future Pyarmor version +* Once CI regfile doesn't work, require new one +* One license can request <= 100 CI regfiles + +.. important:: + + :term:`Pyarmor CI` License doesn't work in local device + + Even in the CI/CD pipeline, :term:`Pyarmor CI` License also doesn't work in the runner which has its own disk. If the runner is not docker container, use :term:`Pyarmor Pro` License instead. -It's no problem to run Pyarmor in Docker or CI pipeline to obfuscate user's application by Pyarmor Basic or Pro license. Register pyarmor with :file:`pyarmor-regfile-xxxx.zip` same as above. **But It's not allowed to distribute pyarmor self and any Pyarmor License to your customers** +.. seealso:: :doc:`ci` -Don't run too many build dockers, maximum is 100 in 24 hours. If more than 100 runs one day, please use Pyarmor Group License. +.. _check device for group license: -And if need run more than 3 build dockers, it's better to start each docker container or runner every 30 seconds, too many register requests in same time may be rejected by license server. +Check Device For Group License +============================== + +Check one device works for group license by this way: + +* First install Pyarmor 8.4.0+ trial version in this device +* Got machine id by the following command:: + + $ pyarmor reg -g 1 + ... + INFO current machine id is "mc92c9f22c732b482fb485aad31d789f1" + INFO device file has been generated successfully + +* Reboot this device, check machine id is same or not +* If machine id is same after each reboot, group license works in this device. Otherwise group license doesn't work in this device. + +For docker container, please check docker host as above. Only if docker host could work with group license, unlimited docker containers could be run in this docker host, refer to :doc:`how-to/register` section ``run unlimited dockers in offline device`` + +**If machine id of docker host is changed after reboot, group license doesn't work in any docker container** + +Most of physics machine, cloud server or VM like qemu, virtual box, vmware with same disk image work with Group license. Most of runners in CI/CD pipeline could not use Group License. + +.. _using group license: Using group license =================== @@ -95,22 +155,19 @@ Using group license Each :term:`Pyarmor Group` could have 100 offline devices, each device has its own number, from 1 to 100. -Only the machine id of device is not changed after reboot, it could be used as group device. Most of physics machine, cloud server or VM like Qemu, Virtual box, Vmware with same disk image work with Group license. If using group in CI pipeline, the default runner doesn't work, but something like `self-host runner`__ may work. +Only the machine id of device is not changed after reboot, it could be used as group device. Most of physics machine, cloud server or VM like Qemu, Virtual box, Vmware with same disk image work with Group license. Refer to :ref:`Check Device For Group License` The allocated device No. is never free, if a device is reinstalled, it need allocate new one. Basic steps: -1. Using activation file :file:`pyarmor-regcode-xxxx.txt` to initial registration, set product name bind to this license, and generate :term:`registration file` [#]_ +1. Using activation file :file:`pyarmor-regcode-xxxx.txt` to initial registration, set product name bind to this license, and generate :term:`registration file` 2. Generating group device file separately on each offline device 3. Using :term:`registration file` and group device file to generate device registration file. 4. Using device registration file to register Pyarmor on offline device [#]_ -.. [#] Pyarmor will review group license manually and enable it in 24 hours since activation file is sent. .. [#] The device registration file is bind to specified device, each device has its own device regfile -__ https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners - Initial registration -------------------- @@ -342,7 +399,17 @@ If there is IPv4 Address, for example ``172.22.32.1``, which is in the same netw Anyway, `pyarmor-auth` must listen on any IPv4 address which is in the same network as docker container. -If there is no available IPv4 address in Windows, the other solution is running `pyarmor-auth` in WSL, in this case, WSL should be taken as offline device. +If there is no available IPv4 address in Windows, the other solution is running `pyarmor-auth` in WSL, in this case, WSL should be taken as offline device. For example:: + + # Create a custom Docker bridge network with a defined subnet + docker network create --subnet=172.17.0.0/16 pyarmor-net + + # Run the container on this network and point host.docker.internal at the gateway + docker run --network pyarmor-net --add-host host.docker.internal=172.17.0.1 ... + +This makes it explicit that the license check requires host and container to be in the same subnet, not just routable. + +Another solution is to run docker with `--network=host`, it make sure docker container has the same netwok netmask with host. **When something is wrong** @@ -376,21 +443,10 @@ Anyway, please configure the docker host/container network so that `pyarmor-auth If run `pyarmor-auth` in Linux VM or WSL, please check group license could work in Linux VM or WSL. It may need generate new device regfile for Linux VM or WSL. -Using group license in CI pipeline ----------------------------------- - -Pyarmor group license could not be used in CI pipeline with default runners, but it may work on something like `self-host runner`__, please check CI documentation for more information. - -The other workaround is that first obfuscating scripts in docker container like above, then create a new branch to store obfuscated scripts in VC server. - -CI pipeline could get obfuscated scripts from this new branch, and start workflow as they're normal Python scripts. - -__ https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners - Using multiple Pyarmor Licenses in same device ============================================== -Generally the registration information is sotred in the Pyarmor :term:`Home Path`, the default value is :file:`~/.pyarmor`. It means +Generally the registration information is stored in the Pyarmor :term:`Home Path`, the default value is :file:`~/.pyarmor`. It means - All Python virtual environments share same registration information - It may not work to register other Pyarmor license in same device @@ -403,6 +459,8 @@ When need many Pyarmor Licenses in one machine, set each license to different pa $ pyarmor --home ~/.pyarmor2 reg pyarmor-regfile-2052.zip $ pyarmor --home ~/.pyarmor2 gen project2/foo.py +.. _pyarmor: + What need to do after upgrading Pyarmor ======================================= @@ -410,18 +468,165 @@ Generally it need do nothing after upgrading Pyarmor, the registration informati But in the following versions something is changed -- **Pyarmor 8.0** Old license for Pyarmor 7 doesn't work +- **Pyarmor 8.0** + + Old license for Pyarmor 7 doesn't work - Some old licenses can be upgraded to Basic License freely, refer to :ref:`upgrade old license ` - Old license can't be upgraded to Pro or Group License -- **Pyarmor 8.6** For Group License it need generate device regfile again with Pyarmor 8.6+. The old device regfile which is generated in prior to Pyarmor 8.6 doesn't work in Pyarmor 8.6+ +- **Pyarmor 9.0** + + A big change about using Pyarmor in CI/CD pipeline + + - :term:`Pyarmor Basic` + + - :ref:`upgrade to pyarmor 9` freely + - If using Pyarmor in CI/CD pipeline, refer to :doc:`ci` + + - :term:`Pyarmor Pro` + + - If not using Pyarmor in CI/CD pipeline, :ref:`upgrade to pyarmor 9` freely + - If using Pyarmor in CI/CD pipeline, 2 choices + + - Still use Pyarmor 8.x as before + - Upgrade to Pyarmor 9, and purchase new :term:`Pyarmor CI` + + - :term:`Pyarmor Group` + + It need generate device regfile for each offline device again by Pyarmor 9.0+, refer to :ref:`upgrade to pyarmor 9` + +- **Pyarmor 9.2** + + From Pyarmor 9.0+ upgrade to 9.2, something is changed for :term:`Pyarmor CI` and :term:`Pyarmor Group`. If Pyarmor version < 9.0, please also check the changes in above **Pyarmor 9.0**. + + - :term:`Pyarmor CI` + + It need request new ci regfile for new version. The old one still works for old version. + + For example, you have one old ci regfile `pyarmor-ci-8000.zip`, use it for old version. + + After upgrade to Pyarmor 9.2+, request new one `pyarmor-ci-8000.zip`:: + + $ pyarmor reg -C pyarmor-regfile-8000.zip + + Then use new one in Pyarmor 9.2+, note that it doesn't work in any prior to versions. + + - :term:`Pyarmor Group` + + It need update device regfile for new version. The old one still works for old version. + + For example, you have old device regfile ``pyarmor-device-regfile-6000.1.zip`` for device no. 1, use it for old version. + + After upgrade to Pyarmor 9.2+, request new one:: + + $ pyarmor reg -g 1 /path/to/pyarmor-regfile-6000.zip + + Then use new one in Pyarmor 9.2+, note that it doesn't work in any prior to versions. + +.. _upgrading old license: + +Upgrading old license +--------------------- + +Not all the old license (Pyarmor 7) could be upgraded to latest version. + +The old license could be upgraded to Pyarmor Basic freely only if it matches these conditions: + +* Following new `Pyarmor EULA`_ +* The license no. starts with ``pyarmor-vax-`` +* The original activation file ``pyarmor-regcode-xxxx.txt`` exists and isn't used more than 100 times +* The old license is purchased before June 1, 2023. In principle, the old license purchased after Pyarmor 8 is available could not be upgraded to new license. + +If failed to upgrade the old license, please purchase new license to use Pyarmor latest version. + +The old license can't be upgraded to Pyarmor Pro and Group. + +**Upgrading old license to Pyarmor Basic** + +First find the activation file ``pyarmor-regcode-xxxx.txt``, which is sent to registration email when purchasing the license. + +Next install Pyarmor 8.2+, according to new `EULA of Pyarmor`_, each license is only for one product. + +Assume this license will be used to obfuscate product ``XXX``, run this command:: + + $ pyarmor reg -u -p "XXX" pyarmor-regcode-xxxx.txt + +Check the upgraded license information:: + + $ pyarmor -v + +After upgrade successfully, do not use activation file ``pyarmor-regcode-xxxx.txt`` again, it's invalid now. A new :term:`registration file` like :file:`pyarmor-regfile-xxxx.zip` will be generated at the same time. + +In other devices using this new :term:`registration file` to register Pyarmor by this command:: + + $ pyarmor reg pyarmor-regfile-xxxx.zip + +After successful registration, all obfuscations will automatically apply this license, and each obfuscation requires online license verification. + +If old license is used by many products (mainly old personal license), only one product could be used after upgrading. For the others, it need purchase new license. + +.. _upgrade to pyarmor 9: + +Upgrade to Pyarmor 9 +-------------------- + +1. :term:`Pyarmor Basic` and :term:`Pyarmor Pro` + + **If Pyarmor License has been registered in this device** + + - First upgrade to Pyarmor 9:: + + $ pip install -U pyarmor + + - When first time to obfuscate scripts, it will show hints:: + + $ pyarmor gen foo.py + + ... + Pyarmor 9 has big change on CI/CD pipeline + If not using Pyarmor License in CI/CD pipeline + Press "c" to continue + Otherwise press "h" to check Pyarmor 9.0 Upgrade Notes + + Continue (c), Help (h), Quit (q): + + - Just press :kbd:`c` to continue, there is no prompt later + + **If Pyarmor License isn't registered in this device** + + - First use :term:`activation file` to generate new :term:`registration file`:: + + $ pip install -U pyarmor + + # Please replace XXX with real product name + $ pyarmor reg -p XXX pyarmor-regcode-xxxx.txt + + - Save and backup new :term:`registration file` ``pyarmor-regfile-xxxx.zip`` + + - Use this new regfile to register Pyarmor in other new device:: + + $ pyarmor reg pyarmor-regfile-xxxx.zip + $ pyarmor -v + + If :term:`activation file` is used too many times, please first install Pyarmor 8, then upgrade to Pyarmor 9 + +2. :term:`Pyarmor Group` License + + It need generate device regfile again with Pyarmor 9.0+ + + - First upgrade to Pyarmor 9:: + + $ pip install -U pyarmor + + - Then generate device regfile as before + + For example, generate device regfile ``pyarmor-device-regfile-6000.1.zip`` for device no. 1:: - - First upgrade Pyarmor to 8.6+ in online device - - Then generate device regfile as first time. For example, generate device regfile ``pyarmor-device-regfile-6000.1.zip`` for device no. 1:: + $ pyarmor reg -g 1 /path/to/pyarmor-regfile-6000.zip - pyarmor reg -g 1 /path/to/pyarmor-regfile-6000.zip + - Finally, use new one to register Pyarmor in offline device:: - - Finally, replace old one with new one + $ pyarmor reg pyarmor-device-regfile-6000.1.zip .. include:: ../_common_definitions.txt diff --git a/docs/how-to/security.rst b/docs/how-to/security.rst index ebbbf5b0..8dcc6406 100644 --- a/docs/how-to/security.rst +++ b/docs/how-to/security.rst @@ -34,6 +34,12 @@ The following options prevent functions or modules from being replaced by hack c * :option:`--assert-call` * :option:`--assert-import` +.. important:: + + If using :option:`--enable-rft`, please also check :doc:`../topic/rftmode`. The default way is simple to use, but may leave more names not changed. If need rename more names, try to use **rft-auto-include** method. + + If using :option:`--enable-bcc`, please also check :doc:`../topic/rftmode` to make sure most of scripts are converted to C code + What's the best performance pyarmor could do? ============================================= @@ -106,4 +112,6 @@ Pyarmor will clear the module level code after the module is imported, the injec But the main script module level code is never cleared, so moving unnecessary code here to another module could improve security. +Note that for `BCC Mode`, the module level code is not converted to C code, so do not move code to module level if using :option:`--enable-bcc`. + .. include:: ../_common_definitions.txt diff --git a/docs/how-to/third-party.rst b/docs/how-to/third-party.rst index 17177f72..795ae989 100644 --- a/docs/how-to/third-party.rst +++ b/docs/how-to/third-party.rst @@ -107,6 +107,9 @@ Here are the list of problem libraries and possible solutions. You are welcome t * - `nuitka`_ - Should work with restrict_module = 0 - Not verify yet + * - `Cython`_ + - Should work with restrict_module = 0 + - .. rubric:: Footnotes @@ -155,8 +158,6 @@ First disable restrict mode:: $ pyarmor cfg restrict_module=0 -Now disable restrict_module, run the nuitka script may raise ``RuntimeError: unauthorized use of script`` - Next use default options to obfuscate the scripts:: $ pyarmor gen foo.py @@ -165,6 +166,10 @@ Finally nuitka the obfuscated script ``dist/foo.py``, check whether it works or Try more options, but I think restrict options such as :option:`--private`, :option:`--restrict`, :option:`--assert-call`, :option:`--assert-import` may not work. +.. note:: + + It may requires v9.0.8+ and non-trial version. Because Nuitka will convert package `pyarmor_runtime_000000/__init__.py` to `pyarmor_runtime_000000_init_.py`, it also results in ``RuntimeError: unauthorized use of script``, this is fixed in v9.0.8 + streamlit --------- @@ -183,4 +188,38 @@ Now obfuscate the scripts:: **It may still not work because of Streamlit may patch code object by itself** +Cython +------ + +Here it's an example show how to `cythonize` a python script `foo.py` obfuscated +by pyarmor:: + + print('Hello Cython') + +First obfuscate it with some extra options:: + + $ pyarmor cfg restrict_module=0 + $ pyarmor gen foo.py + $ ls dist/ + foo.py pyarmor_runtime_000000 + +The obfuscated script and runtime files will be saved in the path `dist` + +Next `cythonize` the obfuscated script `dist/foo.py` to `foo.c`:: + + $ cd dist + $ cythonize -3 foo.py + +Then compile `foo.c` to the extension modules(it may need extra cfalg ``-fPIC`` in some platforms):: + + $ gcc -shared $(python-config --cflags) $(python-config --ldflags) \ + -o foo$(python-config --extension-suffix) foo.c + +Finally test it, remove `dist/foo.py` and import the extension module:: + + $ rm foo.py + $ python -c 'import foo' + +It will print `Hello Cython` as expected. + .. include:: ../_common_definitions.txt diff --git a/docs/index.rst b/docs/index.rst index 0f6106f9..d65d72c6 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -8,6 +8,14 @@ :Authors: |Author| :Copyright: This document has been placed in the public domain. +.. important:: + + New features introduced by Pyarmor 9.1.0 are also in the `Pyarmor V9 Doc`_ + + It includes 3 commands: `pyarmor init`, `pyarmor env`, `pyarmor build` + + And new obfuscated script types: `rft`, `mini`, `vmc`, `ecc` + How the documentation is organized ================================== diff --git a/docs/licenses.rst b/docs/licenses.rst index 3fc77606..c7d50e16 100644 --- a/docs/licenses.rst +++ b/docs/licenses.rst @@ -9,352 +9,374 @@ .. highlight:: console -Introduction -============ - This documentation is only apply to Pyarmor_ 8.0 plus. -Pyarmor is published as shareware, free trial version never expires, but there are some limitations: +Pyarmor is published in `PyPI`_, free trial version never expires. Try it by the following commands:: -(1) Can not obfuscate big scripts -(2) Can not use feature mix-str to obfuscate string constant in scripts -(3) Can not use RFT Mode, BCC Mode -(4) Can not be used for any commercial product. If the total sale income of this product is less than 30x license fees, this software could be used temporarily. -(5) Can not change runtime package name "pyarmor_runtime_000000" -(6) Can not be used to provide obfuscation service in any form, in short this software can't be used to obfuscate the scripts of others -(7) Can not use obf-code > 1 + $ pip install pyarmor + $ pyarmor gen foo.py + $ python dist/foo.py -These limitations can be unlocked by different License Types except (6). All Pyarmor Licenses are only used to obfuscate the scripts written by licensee, in any case they couldn't be used to obfuscate the scripts not written by licensee. +There are some limitations in free version, for example, can't obfuscate big scripts etc. These limitations can be unlocked by different license types. Pyarmor has 4 kind of licenses: -License types -============= +- Basic +- Pro: Irreversible Obfuscation +- Group: Offline build +- CI: for CI/CD pipeline, new in v9.0 + +.. list-table:: Table-1. Compare Different Licenses + :widths: 40 12 12 12 12 12 + :header-rows: 1 + :stub-columns: 1 + + * - Feature + - Free + - Basic + - Pro + - Group + - CI + * - Big Script / Mix String [#]_ + - + - Y + - Y + - Y + - Y + * - BCC / RFT / FLY mode [#]_ + - + - + - Y + - Y + - Y + * - Offline build [#]_ + - + - + - + - Y + - + * - Maximum build devices [#]_ + - + - 100 + - 100 + - 200 + - 0 + * - Unlimited local dockers [#]_ + - + - + - + - Y + - + * - Work in CI/CD pipeline [#]_ + - Y + - Y + - + - + - Y -Pyarmor has 3 kind of licenses: +.. rubric:: Notes -.. glossary:: +.. [#] + - Big Script: file size exceeds a certain value. + - Mix Str: obfuscating string constant in script +.. [#] + - RFT Mode: renaming function/class/method/variable in Python scripts + - BCC Mode: Transforming some Python functions in scripts to c functions, compile them to machine instructions directly +.. [#] Offline build: the build device need not be online to verify Pyarmor License +.. [#] Maximum devices could install Pyarmor, each docker run will be taken as one new build device, CI license only works in CI/CD pipeline +.. [#] Unlimited local dockers: run docker container in local machine, may be offline or in private network +.. [#] Work in CI/CD pipeline: it need special option to register Pyarmor in CI/CD pipeline. - Pyarmor Basic +.. important:: - Basic license could unlock limitations (1) (2) (4) (5) (7). + CI license doesn't work in the runner which has its own disk. - Each obfuscation need online verify license. + If the runner is not docker container, use Pro license instead. - Pyarmor Pro +.. important:: - Pro license could unlock limitations (1) (2) (3) (4) (5) (7). + All Pyarmor Licenses are only used to generate the obfuscated scripts in build device - Each obfuscation need online verify license. + It need neither install Pyarmor nor verify Pyarmor License to execute obfuscated scripts - Pyarmor Group + The obfuscated scripts are generated by Pyarmor, but they are completely independent of Pyarmor - Group license could unlock limitations (1) (2) (3) (4) (5) (7). + You can take them as normal Python scripts, so what the obfuscated scripts do is controlled by you, not by Pyarmor - Group license supports unlimited dockers which use default bridge network and not highly customized, for Pyarmor Basic or Pyarmor Pro, up to 100 docker runs in 24 hours. +Terms of Use +============ - Offline obfuscation, it need not internet connection when obfuscating the scripts. +1. Only use Pyarmor on your own scripts -**For Pyarmor Basic and Pro License, only Pyarmor License file, serial number of hard disk, Ethernet address, IPv4/IPv6 address, and hostname will be sent to remote server for verification. No any user script will be uploaded to remote server** + In any case, even you have purchased Pyarmor License, it's not allowed to obfuscate other scripts of which you haven't property. For example, call pyarmor in your app to obfuscate your customer's scripts, provide obfuscation services based on Pyarmor by website etc. -**Group license is bind to device hardwares, it may not work in any device which hardwares are volatile (docker container is OK only if group license works in docker host)** +2. No profit no license required -**Pyarmor License is only allowed to be installed in the build environments to generate the obfuscated scripts, it need not Pyarmor to run the obfuscated scripts, and it's not allowed to install Pyarmor License with the obfuscated scripts in your customer devices** + Free version can be used to obfuscate your scripts which COULD NOT make lot of money for you. Otherwise it need purchase Pyarmor License. -Each license has an unique number, the format is ``pyarmor-vax-xxxxxx``, which x stands for a digital. +3. One product one license -Each product requires one License No. So any product in global also has an unique number in Pyarmor world. + One product means one kind of product, not one copy of product. For example, Micorsoft Excel is one product, even it's installed on countless devices -**License No. and product name will be embedded into obfuscated scripts, all the other user's information, for example, regname, email are not** + Each license has an unique number, the format is ``pyarmor-vax-xxxxxx``, which x stands for a digital. -If user has many products, and has purchased one license for the first product. The second product could use first product license only if sale income of the second product less than 30x license fees. Once greater than 30x license fees, the second product need purchase its own license. It's same to user's other products. +4. Pay once -One product in Pyarmor world means a product name and everything that makes up this name. + Except CI License, all the other licenses work forever with Pyarmor version when this license is purchased, but **may not work in future Pyarmor versions** -It includes all the devices to develop, build, debug, test product. +5. Fair use -It also includes product current version, history versions and all the future versions. + If you have purchased Pyarmor License for one product, but you have another product, and the total revenue of the other project is less than 100 x Pyarmor License fee, you could rent this license in your aother product -One product may has several variants, each variant name is composed of product name plus feature name. As long as the proportion of the variable part is far less than that of the common part, they're considered as "one product". + Pyarmor CI License has rate limit in CI/CD pipeline: -Pyarmor License could be installed in many machines and devices which belong to licensed product. But there is limitation to be used at the same time. + - 1 run per second + - 100 runs per hour + - 1,000 runs per day + - 10,000 runs per month -In 24 hours only less than 100 devices can use one same Pyarmor Basic or Pro License. Pyarmor License be used means use any feature of Pyarmor in one machine. Running obfuscated scripts generated by Pyarmor is not considered as Pyarmor License be used. +In details check `Pyarmor EULA`_ -For Pyarmor Group License, one license allows 100 offline devices to be registered. The allocated device No. is never free, if a device is reinstalled, it need allocate new one. +.. seealso:: :doc:`Using Pyarmor Licenses ` -.. seealso:: `Pyarmor EULA`_, :doc:`Using Pyarmor Licenses ` +Privacy +======= -**What's one product** +License No. and product name will be embedded into obfuscated scripts, all the other user's information, for example, regname, email are not -First of all, if not for sale, all the Python scripts are belong to one product "non-profits". +For Pyarmor Basic and Pro License, only Pyarmor License file, serial number of hard disk, Ethernet address, IPv4/IPv6 address, and hostname will be sent to Pyarmor License Server for verification -Pyarmor is one product, it includes: +When using Basic or CI License in CI/CD pipeline, some information about docker like docker name, ethernet address, IPv4/IPv6 address, and license information will be sent to Pyarmor License Server for verification -* Pyarmor basic, Pyarmor pro, and Pyarmor group -* pyarmor-webui which provides graphics interface for pyarmor. -* the order system of Pyarmor is a Django's app running in cloud-server. This Django's app also belongs to one product Pyarmor. -* the laptop used to develop Pyarmor, the PCs used to test Pyarmor, the cloud-server to serve order system of Pyarmor, all of them belong to one product Pyarmor. -* Pyarmor 7.x, Pyarmor 8.x and Pyarmor 9.x +No any user script will be uploaded to Pyarmor License Server -Microsoft Office is not one product, because each product in Microsoft Office is functional independence. For example, Microsoft Word and Microsoft Excel belong to Microsoft Office, but they're totally different. +Technical Support +================= -Microsoft Word is one product, and Microsoft Word 2003，Word 2007 etc. are belong to one product Microsoft word. +License fees only for unlock features, not include technical supports -**Hints for Pyarmor licenses** +Users need to learn Pyarmor features and how to use it by themself. Generally Pyarmor Team won't help to debug users' case and teach them how to use Pyarmor -- Basic, Pro and Group license Only work for Python 3.7+ in supported platforms. If using Python 2.7 or prior to 3.7, only Pyarmor old license works. -- If need generate obfuscated scripts offline, only Pyarmor Group or Pyarmor old license works -- Pyarmor Basic and Pro license need internet connection to verify license when obfuscating the scripts, and in 24 hours only 100 different devices are allowed to obfuscate the scripts for each license. -- In 24 hours, if need more than 100 docker containers to obfuscate your scripts, only Pyarmor Group or Pyarmor old license works. -- Pyarmor Basic, Pro and Group licenses don't work for Pyarmor 7. +Pyarmor provides comprehensive learning systems, including but not limited to the following ways: -Check device for group license ------------------------------- +- :doc:`Online documentation ` +- :doc:`Checklist ` and :doc:`FAQs ` could fix 90% issues reported to Pyarmor Team +- `Discussions`_ in Pyarmor project home +- Full examples to show each option usage and common cases by command ``pyarmor man`` +- Learn Pyarmor concepts by figure and animations in EKE Learning Platform (coming soon) + +Rome was not bulit in a day. Pyarmor Team keeps improving documentation and learning systems according to users feedback to make it easy and effects + +Report bugs and request new features in Pyarmor project home, Email to pyarmor@163.com is only for security and private issues, there may no reply for common technical issues. -Check one device works for group license by this way: - -* First install Pyarmor 8.4.0+ trial version in this device -* Got machine id by the following command:: - - $ pyarmor reg -g 1 - ... - INFO current machine id is "mc92c9f22c732b482fb485aad31d789f1" - INFO device file has been generated successfully - -* Reboot this device, check machine id is same or not -* If machine id is same after each reboot, group license works in this device. Otherwise group license doesn't work in this device. - -For docker container, please check docker host as above. Only if docker host could work with group license, unlimited docker containers could be run in this docker host, refer to :doc:`how-to/register` section ``run unlimited dockers in offline device`` - -**If machine id of docker host is changed after reboot, group license doesn't work in any docker container** - -Most of physics machine, cloud server or VM like qemu, virtual box, vmware with same disk image work with Group license. If using group in CI pipeline, the default runner doesn't work, but something like `self-host runner`__ may work. - -__ https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners - -License features ----------------- - -.. table:: Table-1. License Features - :widths: auto - - =================== ======== ======== ========= ======== ============== - Features Trial Basic Pro Group Remark - =================== ======== ======== ========= ======== ============== - Basic Obfuscation Y Y Y Y [5]_ - Expired Script Y Y Y Y [6]_ - Bind Device Y Y Y Y [7]_ - JIT Protection Y Y Y Y [8]_ - Assert Protection Y Y Y Y [9]_ - Themida Protection Y Y Y Y [10]_ - Big Script No Y Y Y [1]_ - Mix Str No Y Y Y [2]_ - obf-code > 1 No Y Y Y [11]_ - RFT MODE No No Y Y [3]_ - BCC MODE No No Y Y [4]_ - Unlimited dockers Y No No Y [12]_ - =================== ======== ======== ========= ======== ============== - -.. rubric:: notes - -.. [1] Big Script means file size exceeds a certain value. -.. [2] Mix Str: obfuscating string constant in script -.. [3] RFT Mode: renaming function/class/method/variable in Python scripts -.. [4] BCC Mode: Transforming some Python functions in scripts to c functions, compile them to machine instructions directly -.. [5] Basic Obfuscation: obfuscating the scripts by default options -.. [6] Expired Script: obfuscated scripts has expired date -.. [7] Bind Device: obfuscated scripts only run in specified devices -.. [8] JIT Protection: processing some sensitive data by runtime generated binary code -.. [9] Assert Protection: preventing others from hacking obfuscated scripts -.. [10] Themida Protection: using Themida to protect Windows dlls -.. [11] ``--obf-code=2`` is new in Pyarmor 8.2 -.. [12] This feature is introduced in Pyarmor 8.3, group license supports unlimited dockers, basic and pro licenses only allow 100 runs one day. Also check above section ``Check device for group license`` +Pyarmor team generally will handle submitted issues within 24 hours (working time), but may be extended during holidays or special circumstances + +Pyarmor Team doesn't provide any instant technical support by telphone or other similar tools. Purchasing license ================== -Open shopping cart in any web browser: +If you have Pyarmor 8.6+ installed, this command also could open shopping cart:: - https://order.mycommerce.com/product?vendorid=200089125&productid=301044051 + $ pyarmor reg --buy -If you have Pyarmor 8.0+ installed, this command also could open shopping cart:: +Open `Pyarmor Shopping Cart`_ in any web browser: - $ pyarmor reg --buy + https://jondy.github.io/paypal/index.html -In the shopping cart, select License Type and complete the payment online. +All of these license are only for Pyarmor 8.0+ with Python 3.7+, if need work with Pyarmor 7.x which supports Python < 3.7, please purchase `Pyarmor Old License`: -Please fill reg-name with personal or company name when placing order. + https://jondy.github.io/paypal/obsolete.html -.. list-table:: Table-2. License Prices - :header-rows: 1 +Refund policy +============= - * - License Type - - Net Price($) - - Remark - * - Basic - - 52 - - - * - Pro - - 89 - - - * - Group - - 158 - - +If activation file isn't used, and purchasing date is in 30 days, refund is acceptable. -An activation file named ``pyarmor-regcode-xxxx.txt`` will be sent by email immediately after payment is completed successfully. +- If purchasing order from reseller, contact your reseller -Following the guide in activation file to take the purchased license effects, or check :doc:`../how-to/register` +- For other cases, email to pyarmor@163.com -**There are no additional license fees, apart from the cost of the license. And it only needs to be paid once, not periodically** +- Pyarmor Team may charge 5% fee for some refund cases. -**The license is valid forever in Pyarmor version when purchasing it, but may not work in Pyarmor future versions** +Out of 30 days, or activation file has been used, refund request will be rejected. -.. important:: +Appendix +======== - License fee is only for unlocking features, not for technical support. +What is one product +------------------- - Refer to :ref:`asking questions` for technical support. +First of all, if not for sale, all the Python scripts are belong to one product "non-profits". -Refund policy -------------- +One product in Pyarmor world means a product name and everything that makes up this name. -If activation file isn't used, and purchasing date is in 30 days, refund is acceptable. Please +It includes all the devices to develop, build, debug, test product. -1. Email to Ordersupport@mycommerce.com with order information and ask for refund. -2. Or click `FindMyOrder page`_ to submit refund request +It also includes product current version, history versions and all the future versions. -Out of 30 days, or activation file has been used, refund request will be rejected. +One product may has several variants, each variant name is composed of product name plus feature name. As long as the proportion of the variable part is far less than that of the common part, they're considered as "one product". -.. _FindMyOrder page: https://www.findmyorder.com/store?Action=DisplayEmailCustomerServicePage&Env=BASE&Locale=en_US&SiteID=findmyor +Pyarmor is one product, it includes: +* Pyarmor basic, Pyarmor pro, and Pyarmor group +* pyarmor-webui which provides graphics interface for pyarmor. +* the order system of Pyarmor is a Django's app running in cloud-server. This Django's app also belongs to one product Pyarmor. +* the laptop used to develop Pyarmor, the PCs used to test Pyarmor, the cloud-server to serve order system of Pyarmor, all of them belong to one product Pyarmor. +* Pyarmor 7.x, Pyarmor 8.x and Pyarmor 9.x -.. _upgrading old license: +Microsoft Office is not one product, because each product in Microsoft Office is functional independence. For example, Microsoft Word and Microsoft Excel belong to Microsoft Office, but they're totally different. -Upgrading old license -===================== +Microsoft Word is one product, and Microsoft Word 2003，Word 2007 etc. are belong to one product Microsoft word. -Not all the old license could be upgraded to latest version. +.. _select-license-type: -The old license could be upgraded to Pyarmor Basic freely only if it matches these conditions: +Which license type is right for me +---------------------------------- -* Following new `Pyarmor EULA`_ -* The license no. starts with ``pyarmor-vax-`` -* The original activation file ``pyarmor-regcode-xxxx.txt`` exists and isn't used more than 100 times -* The old license is purchased before June 1, 2023. In principle, the old license purchased after Pyarmor 8 is available could not be upgraded to new license. +All of the following licenses are only for Python 3.7+ -If failed to upgrade the old license, please purchase new license to use Pyarmor latest version. +.. list-table:: Table-2. Select Different Licenses + :widths: 40 12 12 12 12 12 + :header-rows: 1 + :stub-columns: 1 + + * - Condition + - Free + - Basic + - Pro + - Group + - CI + * - Less than 100 runs per month in CI/CD pipeline + - Y + - Y + - Y + - + - Y + * - More than 100 runs per month in CI/CD pipeline + - Y + - Y + - + - + - Y + * - Need offline obfuscation + - Y + - + - + - Y + - + * - Need irreversible obfuscation + - + - + - Y + - Y + - Y + * - Less than 100 runs per month in local dockers + - Y + - Y + - Y + - Y + - Y + * - More than 100 runs per month in local dockers + - Y + - Y + - + - Y + - Y -The old license can't be upgraded to Pyarmor Pro and Group. +.. _how-many-licenses-required: -.. - But there is discount for old users to purchase Pyarmor Pro. +How many licenses are required +------------------------------ - .. list-table:: Table-3. Pyarmor Pro Discount for Old Users - :header-rows: 1 +1. List all the products which are sold separately - * - Old License Purchased Date - - Discount(%) - - Remark - * - Between 2022-06-01 and 2023-05-31 - - 40 - - - * - Between 2021-06-01 and 2022-05-31 - - 30 - - - * - Before 2021-05-31 - - 20 - - + - If the sales revenue of the product is less than 100 x Pyarmor license fee, there is no need to list the product + - If no more than 2 products left, one license is OK - Please send your order no. to pyarmor@163.com to request discount coupon, this request must be sent from registration email. +2. How to tell diffenent product could be taken as one proudct in Pyarmor view -Upgrading old license to Pyarmor Basic --------------------------------------- + Suppose there are 2 products in step 1: X and Y -First find the activation file ``pyarmor-regcode-xxxx.txt``, which is sent to registration email when purchasing the license. + a. case 1, X and Y could use one license -Next install Pyarmor 8.2+, according to new `EULA of Pyarmor`_, each license is only for one product. + - Y includes whole X features + - The extra features of Y is related to X features -Assume this license will be used to obfuscate product ``XXX``, run this command:: + For example, Pyarmor Basic (X), Pyarmor Pro (Y) could use one license because - $ pyarmor reg -u -p "XXX" pyarmor-regcode-xxxx.txt + - Pyarmor Pro includes all the features of Pyarmor Basic + - The extra features of Pyarmor Pro is irreversible obfuscation which is an enhancement of Pyarmor Basic -Check the upgraded license information:: + b. case 2, X and Y need 2 licenses - $ pyarmor -v + - Y includes whole X features + - But X features is very small in the Y features -After upgrade successfully, do not use activation file ``pyarmor-regcode-xxxx.txt`` again, it's invalid now. A new :term:`registration file` like :file:`pyarmor-regfile-xxxx.zip` will be generated at the same time. + For example, X is a facial recognition product, B is an attendance management system that uses facial recognition functionality -In other devices using this new :term:`registration file` to register Pyarmor by this command:: + c. case 3, X and Y could use one license - $ pyarmor reg pyarmor-regfile-xxxx.zip + - Y is a functional supplement for product X -After successful registration, all obfuscations will automatically apply this license, and each obfuscation requires online license verification. + For example, X is CAD Editor, Y is CAD Tool which is used to convert CAD file to PDF file -If old license is used by many products (mainly old personal license), only one product could be used after upgrading. For the others, it need purchase new license. + d. case 4, X and Y need 2 licenses -.. - Upgrading old license to Pyarmor Pro - ------------------------------------ + - The functions of X and Y are almost independent - Upgrading old license to Pyarmor Pro needs extra fees. + For example, Microsoft Word (X), Micorsoft Excel (Y) are 2 products, even they're belong to Micorsoft Office Suite - .. list-table:: Table-3. Upgrade fee from old license - :header-rows: 1 +3. Regard to shared backend system - * - License Type - - Upgrading fee($) - - Remark - * - Basic - - 0 - - following new EULA and match some conditions - * - Pro - - 50 - - - * - Group - - N/A - - + There are 2 product A and B, share one backend engineer C - Open shopping cart in any web browser: + a. case 1: need obfuscate backend C, but frontend of A and B need not - https://order.mycommerce.com/product?vendorid=200089125&productid=301044051 + - one license for backend C is OK - If you have Pyarmor 8.2+ installed, this command also could open shopping cart:: + b. case 2: need obfuscate backend C, and the frontend of A and B - $ pyarmor reg --buy + - 2 licenses, one for A, another for B + - C need not new license, use any license for A or B - In the shopping cart, select ``Pyarmor-upgrade`` and complete the payment online. +4. Many products need use one license in technical - A file named ``pyarmor-regcode-to-pro.txt`` will be sent by email immediately after payment is completed successfully. + If there are many products (in Pyarmor view) need use one Pyarmor license in development view, it should - This file includes the guide to upgrade old license to Pyarmor Pro. + - Purchase many licenses + - Register each product with different license once + - Use one of registeration file to generate the obfuscated scripts for all the products - Internet connection and Pyarmor 8.2+ are required to upgrade old license, check Pyarmor version by this command +.. _how-to-upgrade-license: - pyarmor -v +How to upgrade Pyarmor Licenses +------------------------------- - Check old license +Now it doesn't support to upgrade one Pyarmor License to any other Pyarmor License. For example, upgrade Pyarmor Basic to Pyarmor Pro by paying price difference - pyarmor-7 -v +There are only the following special cases - If no old license found, register old license first +- Pyarmor Old License used by Pyarmor <= 7.x could be upgrade to Pyarmor Basic License in some conditions +- If upgrading Pyarmor v8 to v9, Pyarmor License need to be re-activated - pyarmor-7 register pyarmor-regcode-xxxx.txt +Please refer to `What need to do after upgrading Pyarmor `_ - Assume this license will be used to obfuscate product ``XXX``, run this command:: +In short, if it could be upgraded successfully by the guide in the documentation, it's OK. If something is wrong, this license can't be upgraded. Pyarmor Team doesn't handle this kind of request, there may be no any reply. - $ pyarmor reg -u -p "XXX" pyarmor-regcode-xxxx.txt +Q & A +----- - Check the upgraded license information:: +**I have just started selling my product, do I can use Pyarmor trial version to product it** - $ pyarmor -v +Before the value of sales exceed 100 x Pyarmor license fee, Pyarmor trial version can be used to obfuscate the product. After the value of sales exceed, it need purchase Pyarmor license. - After upgrade successfully, the file ``pyarmor-regcode-to-pro.txt`` is invalid. A new :term:`registration file` like :file:`pyarmor-regfile-xxxx.zip` will be generated at the same time. +**I noticed that the pricing table lists "0 Maximum build devices," for the CI/CD plan which has left me a bit confused. Could you please clarify what this means?** - In other devices using this new :term:`registration file` to register Pyarmor by this command:: + It means CI License only works in CI/CD pipeline, can’t be used in local device. - $ pyarmor reg pyarmor-regfile-xxxx.zip +**If we are using the product in our CI/CD environment we can deploy it to unlimited number of kubernetes pods** - After successful registration, all obfuscations will automatically apply this license, and each obfuscation requires online license verification. + If the product means the obfuscated scripts, Pyarmor has no any limitation on it. - If old license is used by many products (mainly old personal license), only one product could be used after upgrading. For the others, it need purchase new license. + Pyarmor License is only apply to build machine in which to generate the obfuscated scripts. .. include:: _common_definitions.txt diff --git a/docs/part-2.rst b/docs/part-2.rst index 23db3c53..e4eee10d 100644 --- a/docs/part-2.rst +++ b/docs/part-2.rst @@ -10,8 +10,8 @@ how-to/packing how-to/wheel how-to/obfuscation + how-to/register how-to/ci how-to/third-party - how-to/register .. include:: _common_definitions.txt diff --git a/docs/questions.rst b/docs/questions.rst index 4e8cb32c..6ebf6167 100644 --- a/docs/questions.rst +++ b/docs/questions.rst @@ -12,7 +12,7 @@ If using :command:`pyarmor-7` or Pyarmor < 8.0, please check `Pyarmor 7.x Doc`_ .. important:: - Pyarmor team handles too many wrong usage issues, so one document :doc:`reference/solutions` has been arranged to solve this kind of issue quickly. If you aren't sure this issue is wrong usage or not, please check this doc at first. + Pyarmor team handles too many wrong usage issues, so one document :doc:`reference/solutions` has been arranged to solve this kind of issue quickly. If you aren't sure this issue is wrong usage or not, please check this doc or https://pyarmor.eke.org.cn/ at first. Pyarmor team will mark this kind of issue as `invalid` or `documented` and close it immediately. @@ -67,7 +67,7 @@ A good report should have - Actual results - Expected results -For different issues, please following different guide to report bug, and do not paste snapshot image but paste text directly. +It's recommand to report issue by command `pyarmor man`. .. important:: @@ -78,34 +78,18 @@ Build issues If there is error message when run pyarmor, please first check :doc:`reference/errors` to find solutions -If still no solution, please run pyarmor with debug option :option:`-d`. For example:: +If still no solution, please report issue based on :file:`pyarmor.bug.log` generated by Pyarmor automically. For example:: - $ pyarmor -d gen ... - $ pyarmor -d reg ... + [BUG]: no found input "fooxxx.py" -It will generate file :file:`pyarmor.report.bug` like this:: + ## Command Line + pyarmor gen fooxxx.py - [Bug] `FileNotFoundError: [Errno 2] No such file or directory: 'aa.zip'` - - ### Full command options and console output - pyarmor -d reg aa.zip - - 2024-05-30 21:50:52,682 Python 3.7.10 - 2024-05-30 21:50:52,684 Pyarmor 8.5.9 (pro), 005068, btarmor - 2024-05-30 21:50:52,696 Platform darwin.x86_64 - 2024-05-30 21:50:52,696 native platform darwin.x86_64 - 2024-05-30 21:50:52,696 home path: /Users/jondy/.pyarmor - 2024-05-30 21:50:52,696 register "aa.zip" - 2024-05-30 21:50:52,698 unknown error, please check pyarmor.error.log - 2024-05-30 21:50:52,704 [Errno 2] No such file or directory: 'aa.zip' - - ### Traceback - Traceback (most recent call last): - ... - self.fp = io.open(file, filemode) - FileNotFoundError: [Errno 2] No such file or directory: 'aa.zip' - -Take the first line `[Bug] ...` as bug title, and the rest content as bug body and make necessary supplements and explanations + ## Environments + Home /Users/jondy/.pyarmor + Platform darwin.x86_64 (darwin.x86_64) + Python 3.12.0 + Pyarmor 9.0.4 (group), 006000, btarmor Pack issues ----------- @@ -295,23 +279,9 @@ Hot Questions License ======= -**Which license is right for my project?** - - Hints for all licenses: - - - If using Python 2.7 or Python <= 3.6, then only Pyarmor old license works. All the new licenses only work for Python 3.7+ - - Each license only need pay once, but it may not work in Pyarmor future versions - - Hints for 3 new licenses (Pyarmor Basic, Pro and Gropu license): - - - If need offline obfuscation, only Pyarmor Group license works - - Pyarmor Basic and Pro license need internet connection to verify license when obfuscating the scripts, and in 24 hours only 100 different devices are allowed for each license - - In 24 hours, if need more than 100 docker containers to obfuscate your scripts, only Pyarmor Group license works. - - Pyarmor Basic, Pro and Group licenses don't work for Pyarmor 7. - **Will Pyarmor Pro license upload my scripts to remote server to verify license?** -No. For Pyarmor Basic and Pro License, only Pyarmor License file, serial number of hard disk, Ethernet address, IPv4/IPv6 address, and hostname will be sent to remote server for verification. + No. For Pyarmor Basic and Pro License, only Pyarmor License file, serial number of hard disk, Ethernet address, IPv4/IPv6 address, and hostname will be sent to remote server for verification. **I am interested to know if the users are entitled to updates to ensure compatibility with future versions of Python.** @@ -319,7 +289,7 @@ No. For Pyarmor Basic and Pro License, only Pyarmor License file, serial number **we use Docker to build/obfuscate the code locally then publish the Docker file to the client. After the build stage, the whole environment (and the license) is gone. I wonder how the workflow would be? Can I add the license file to the pipeline and register every time and build?** - It's no problem to run Pyarmor in Docker or CI pipeline to obfuscate application. Each build registering pyarmor with :file:`pyarmor-regfile-xxxx.zip` which is generated in initial registration. But It's not allowed to distribute package pyarmor and :term:`Pyarmor Basic`, :term:`Pyarmor Pro`, :term:`Pyarmor Group` License to customer, and don't run too many build dockers. + Please refer to :doc:`how-to/ci` **We are currently using a trial license for testing, but unfortunately our scripts are big and we are not able to statistically test the operation of Pyarmor. Do you have a commercial trial license for a certain trial period so that we can test the operation of Pyarmor for our scripts?** @@ -345,9 +315,15 @@ No. For Pyarmor Basic and Pro License, only Pyarmor License file, serial number **Which PyArmor 8.0 license for CI, more than 100 runs / day** - At this time, only Group License with self-host runner may work, the default runner which has volatile hardwares may not work. For CI with group license, there are 2 workarouds, please check section `using group license in CI pipeline`__ + It's recommend to upgrade to Pyarmor 9, and use :term:`Pyarmor CI` License or :term:`Pyarmor Basic` License. See also :doc:`how-to/ci` + +**We should be able to assume that the CI regfile will keep working as long as it is within the license limits. Otherwise, builds might break at a moment notice. Could you confirm whether it is safe to assume that the CI license will keep working?** + + Fix Pyarmor version in the CI/CD pipeline, CI regfile works within the validity period. -__ https://pyarmor.readthedocs.io/en/latest/how-to/register.html#using-group-license-in-ci-pipeline +**About CI License, is there an option to allow offline usage in the CI/CD pipeline?** + + No. There is no offline option for CI License. .. **We intend to use PyArmor in CI to build obfuscated Docker images. According to the docs we can't use PyArmor in CI because the machine IDs will be different across each CI run (we verified this is the case), but according to this section in the docs, we can use PyArmor on "all the devices to develop, build, test the product", so we're a bit confused on whether this would work in CI. Could we buy a group license and then activate PyArmor as part of the CI pipeline? All of these different CI machines and developer laptops are building "one product"** @@ -365,7 +341,7 @@ Upgrading The old license is valid for ever. In this case need not upgrade old license to Pyarmor Basic license, just install Pyarmor 8.x, and using :command:`pyarmor-7` with old license. - Check :doc:`licenses` for more information about upgrading + Check :doc:`how-to/register` for more information about upgrading **If we upgrade the old license, will the current license expire? (no more available in terms of Pyarmor v7?** @@ -395,6 +371,10 @@ Upgrading There is no identical pack in Pyarmor 8, Pyarmor 8+ only provide repack function to handle bundle of PyInstaller. Refer to basic tutorial, topic `insight into pack`__ and this solved issue `Pyarmor pack missing in pyarmor 8.0`__ +**Using PyArmor 9.x newer version, we cannot generated licenses compatible with "Software" created using older PyArmor (7.x versions), mainly using license.lic. Correct?** + + You're right. At this time one possible solution is still using Pyarmor 7 obfuscated script to verify old runtime key, in Pyarmor 9 obfuscated script check old runtime key by calling Pyarmor 7 obfuscated script indirectly (IPC) + __ https://pyarmor.readthedocs.io/en/stable/topic/repack.html __ https://github.com/dashingsoft/pyarmor/discussions/1107 @@ -405,8 +385,14 @@ Purchasing If this order isn't activated and in 30 days since purchasing, you can refund the order by one of ways - 1. Email to Ordersupport@mycommerce.com with order information and ask for refund. - 2. Or click `FindMyOrder page`_ to submit refund request + * If purchasing order from MyCommerce: + + 1. Email to Ordersupport@mycommerce.com with order information and ask for refund. + 2. Or click `FindMyOrder page`_ to submit refund request + + * If purchasing order from reseller, contact your reseller + + * For other cases, email to pyarmor@163.com .. _FindMyOrder page: https://www.findmyorder.com/store?Action=DisplayEmailCustomerServicePage&Env=BASE&Locale=en_US&SiteID=findmyor diff --git a/docs/reference/concepts.rst b/docs/reference/concepts.rst index 068a3555..2c7611a7 100644 --- a/docs/reference/concepts.rst +++ b/docs/reference/concepts.rst @@ -105,6 +105,10 @@ A :term:`Pyarmor License` type + Pyarmor CI + + A :term:`Pyarmor License` type + Pyarmor Group A :term:`Pyarmor License` type diff --git a/docs/reference/environments.rst b/docs/reference/environments.rst index f981ef4d..71489193 100644 --- a/docs/reference/environments.rst +++ b/docs/reference/environments.rst @@ -20,14 +20,14 @@ Supported Python versions .. table:: Table-1. Supported Python Versions :widths: auto - =================== ===== ========= ========== ====== ====== ======= ============== - Python Version 2.7 3.0~3.6 3.7~3.10 3.11 3.12 3.13+ Remark - =================== ===== ========= ========== ====== ====== ======= ============== - pyarmor 8 RFT Mode No No Y Y Y N/y [#]_ - pyarmor 8 BCC Mode No No Y Y Y N/y - pyarmor 8 others No No Y Y Y N/y - pyarmor-7 Y Y Y No No No - =================== ===== ========= ========== ====== ====== ======= ============== + =================== ===== ========= ========== ============ ============== + Python Version 2.7 3.0~3.6 3.7~3.10 3.11~3.15 Remark + =================== ===== ========= ========== ============ ============== + pyarmor 8 RFT Mode No No Y Y [#]_ + pyarmor 8 BCC Mode No No Y Y + pyarmor 8 others No No Y Y + pyarmor-7 Y Y Y No + =================== ===== ========= ========== ============ ============== Supported platforms =================== @@ -36,7 +36,7 @@ Supported platforms :widths: auto =================== ============ ======== ======= ============ ========= ======= ======= - OS Windows Apple Linux [#]_ + OS Windows Apple [#]_ Linux [#]_ ------------------- ------------ ----------------- ----------------------------------------- Arch x86/x86_64 x86_64 arm64 x86/x86_64 aarch64 armv7 armv6 =================== ============ ======== ======= ============ ========= ======= ======= @@ -78,6 +78,7 @@ Supported platforms .. rubric:: notes .. [#] ``N/y`` means not yet now, but will be supported in future. +.. [#] Apple Silcon only supports Python 3.9+ .. [#] This Linux is built with glibc .. [#] pyarmor-7 also supports more linux arches, refer to `Pyarmor 7.x platforms`__. .. [#] These platforms are introduced in Pyarmor 8.3 @@ -302,6 +303,15 @@ Check it, first copy this script to ``.pyarmor/hooks/pyarmor_runtime.py``:: ... RuntimeError: unauthorized use of script (1:10325) +If need query hardware information, the simple way is to import Pyarmor extension `pytransform3`. For example, in Windows, copy the corresponding `pytransform3.pyd` to target device, then get machine id by this way + +.. code-block:: python + + def bootstrap(user_data): + from pytransform3 import get_hd_info + # Refer to pyarmor/cli/get_hd_info.py + print('Machine ID: %s' % get_hd_info(22).decode()) + .. _target environments: ===================== diff --git a/docs/reference/errors.rst b/docs/reference/errors.rst index 6a90a475..a3ef4e13 100644 --- a/docs/reference/errors.rst +++ b/docs/reference/errors.rst @@ -70,7 +70,9 @@ Building Errors Please use ``pyarmor-7`` or downgrade pyarmor to 7.7.4 * - This code has been used too many times - - If this code is used in CI/Docker pipeline, please send **order information** by registration email of this code to pyarmor@163.com to unlock it. Do not send this code only, it doesn't make sense. + - It has been locked and can't be used again. + * - no registration code found in pyarmor-regcode-xxxx.txt + - Download `pyarmor-regcode-xxxx.txt` again, check its content, make sure it's same as email body * - update license token failed - If run register command more than 3 times in 1 minute, wait for 5 minutes, and try again. @@ -91,6 +93,23 @@ Building Errors b'NO:missing parameter' If not return this, but raises exception, it's firewall problem, please configure it to allow Python interpreter to visit `pyarmor.dashingsoft.com:80` + * - invalid activation file + - Make sure it's the attachment of activation email which title likes `[Pyarmor] License Activation Code` + + Make sure it's not interpolated by email server or email client. + + Make sure its encoding is utf-8. + + It should be same as the email content which starts with `Dear ...` + + It should include the activation code in one line (it's important there should no line break in the activation code). + + * - HTTP Error 400 - no shipping information + - Generally something tamper with the activation code sent by Pyarmor Team. + + Maybe the email client, maybe some auto-fix typos tools or anti-virus tools touch the email content. + + Try to check the original email from web portal in the web-browser to verify the activation code. Runtime Errors ============== @@ -212,4 +231,20 @@ By searching error message in google or any other search engine to find the solu __ https://github.com/dashingsoft/pyarmor/discussions/1674 +**No such file or directory: 'nul'** + +Generally something is wrong with Windows System. + +Try to create NUL device again by this command:: + + sc create null binpath=C:\Windows\System32\drivers\null.sys type=kernel start=auto error=normal + +Then start null:: + + sc start null + +If it works, this problem should be fixed. + +If not, please google the solution. It's Windows System issue. + .. include:: ../_common_definitions.txt diff --git a/docs/reference/man.rst b/docs/reference/man.rst index 5e3d7f8b..17aefa4d 100644 --- a/docs/reference/man.rst +++ b/docs/reference/man.rst @@ -25,7 +25,7 @@ pyarmor -h, --help show available command set then quit -v, --version show version information then quit -q, --silent suppress all normal output :option:`... <-q>` --d, --debug show more information in the console :option:`... <-d>` +-d, --debug generate debug log file :option:`... <-d>` --home PATH set Pyarmor HOME path :option:`... <--home>` These options can be used after :program:`pyarmor` but before command, here are available commands: @@ -51,11 +51,12 @@ For example:: .. option:: -d, --debug - Show more information in the console + Generate debug log :file:`pyarmor.debug.log` -When something is wrong, print more debug information in the console. For example:: +When something is wrong, use this option to generate :file:`pyarmor.debug.log` to get more information. For example:: - pyarmor -d gen foo.py + $ pyarmor -d gen foo.py + $ cat pyarmor.debug.log .. option:: --home PATH[,GLOBAL[,LOCAL[,REG]]] @@ -337,11 +338,11 @@ Using this options multiple times means binding many machines. For example, the In case there are more network cards, binding anyone by this form:: - $ pyarmor gen -b "<2a:33:50:46:8f>" foo.py + $ pyarmor gen -b "*IFMAC:<2a:33:50:46:8f>" foo.py Bind all network cards by this form:: - $ pyarmor gen -b "<2a:33:50:46:8f,f0:28:69:c0:24:3a>" foo.py + $ pyarmor gen -b "*IFMAC:<2a:33:50:46:8f,f0:28:69:c0:24:3a>" foo.py In Linux, it's possible to bind named Ethernet card:: @@ -356,6 +357,18 @@ In Linux, binding to specify name:: $ pyarmor gen -b "/dev/vda2:KDX3298FS6P5AX380" foo.py +It also supports prefix for each hardware type, for example:: + + $ pyarmor gen -b "*HARDDISK:0000_1111_2222_3333 *IFMAC:11:22:33:44:55:66" + +This could fix some runtime key failed issues. All supported prefix are + +- ``*MID:`` +- ``*IFMAC:`` +- ``*IFIPV4:`` +- ``*DOMAIN:`` +- ``*HARDDISK:`` + .. option:: --bind-data DATA DATA may be ``@FILENAME`` or string @@ -366,6 +379,8 @@ If DATA has a leading ``@``, then the rest is a filename. Pyarmor reads the bina For any other case, DATA is converted to bytes as private data. +By special builtin function :func:`__pyarmor__`, the bind data can be read in the obfuscated scripts. Refer to this example `Using hook to bind script to docker ` in the :doc:`../tutorial/customization` + .. option:: --period N Check :term:`Runtime Key` periodically. @@ -543,13 +558,15 @@ Use :term:`JIT` to process some sensitive data to improve security. Mix the string constant in scripts :sup:`basic` + This option doesn't touch any docstring + It may reduce performance if there are too many strings, in this case, only mix important strings by filter. .. seealso:: `Filter mix string` in :doc:`../tutorial/advanced` .. option:: --assert-call - Assert function is obfuscated + Assert function is obfuscated, this option implies :option:`--private`. If this option is enabled, Pyarmor scans each function call in the scripts. If the called function is in the obfuscated scripts, protect it as below, and leave others as it is. For example, @@ -581,7 +598,7 @@ In this example, ``fib`` is protected, ``print`` is not. .. option:: --assert-import - Assert module is obfuscated + Assert module is obfuscated, this option implies :option:`--private`. If this option is enabled, Pyarmor scans each ``import`` statement in the scripts. If the imported module is obfuscated, protect it as below, and leave others as it is. For example, @@ -1018,6 +1035,33 @@ Not all the old license could be upgrade to new license, check :doc:`../licenses Valid value is from 1 to 100 +.. _pyarmor init: + +pyarmor init +============ + +.. versionadded:: 9.1 + +Refer to `Pyarmor V9 Man`_ + +.. _pyarmor env: + +pyarmor env +=========== + +.. versionadded:: 9.1 + +Refer to `Pyarmor V9 Man`_ + +.. _pyarmor build: + +pyarmor build +============= + +.. versionadded:: 9.1 + +Refer to `Pyarmor V9 Man`_ + Environment Variables ===================== diff --git a/docs/reference/solutions.rst b/docs/reference/solutions.rst index b3051668..e266b729 100644 --- a/docs/reference/solutions.rst +++ b/docs/reference/solutions.rst @@ -13,6 +13,8 @@ Build Device If something is wrong in build device, check this section. +.. _fix-bootstrap-issue: + Bootstrap failed ---------------- @@ -24,7 +26,7 @@ Bootstrap failed 2. Check extension `pytransform3` is exact for this Python and platform - Test it by Python interperter: + Test it by Python interpreter: .. code-block:: bash @@ -88,13 +90,22 @@ Bootstrap failed $ export PYARMOR_PLATFORM=linux.loongarch64 $ pyarmor gen foo.py -4. If this platform is supported, try to upgrade Python interprerter to latest patch version. For example, upgrade Python 3.11.0b2 to 3.11.9 +4. If this platform is supported, try to upgrade Python interpreter to latest patch version. For example, upgrade Python 3.11.0b2 to 3.11.9 + +.. _fix-register-issue: Registration Failed ------------------- If it's using :term:`Activation File` (`pyarmor-regcode-xxxx.txt`), make sure this file is not used more than 3 times, generally once initial registration completed, activation file :file:`pyarmor-regcode-xxxx.txt` is invalid. It should use :term:`Registration File` ``pyarmor-regfile-xxxx.zip`` for any next registration. +**Error: invalid activation file** + +It means that the activation file is not issued by Pyarmor Team, please make sure: + +- It's exactly the attachment of activation email which title is `[Pyarmor] License Activation Code` and this attchment is NOT modified by your mail server +- The content of activation file should be same as the email content which starts with `Dear ...` and includes the activation code in one line (it's important there is no line break in the activation code). + **Basic/Pro License** 1. If the date time of this device has been changed, restore it to current time @@ -111,7 +122,7 @@ If it's using :term:`Activation File` (`pyarmor-regcode-xxxx.txt`), make sure th Otherwise check network configuration -6. Check Python interpreter by the following commands (If there are many Python installed, make sure this Python interperter is used to execute Pyarmor) +6. Check Python interpreter by the following commands (If there are many Python installed, make sure this Python interpreter is used to execute Pyarmor) .. code-block:: bash @@ -121,7 +132,7 @@ If it's using :term:`Activation File` (`pyarmor-regcode-xxxx.txt`), make sure th >>> print(res.read()) b'NO:missing parameter' -If it raises exception or return something else, it’s firewall problem, please configure firwire to allow Python interpreter to visit pyarmor.dashingsoft.com and port 80 or 443 +If it raises exception or return something else, it’s firewall problem, please configure firewall to allow Python interpreter to visit pyarmor.dashingsoft.com at port 80 or 443 If it returns as above, but still failed to register, report issue with license no. like `pyarmor-vax-5068` @@ -147,6 +158,8 @@ If it returns as above, but still failed to register, report issue with license 1. Check docker host and container network, make sure they're in same network +.. _fix-obfuscate-issue: + Obfuscation failed ------------------ @@ -175,10 +188,12 @@ Please check above section `Registration Failed` - A script as simple as possible to reproduce issue - Do not use the options which doesn't make sense for this issue +.. _fix-pack-issue: + Packing failed -------------- -1. Try to pack the original script by PyInstaller directly first, mkae sure it works and the final bundle works +1. Try to pack the original script by PyInstaller directly first, make sure it works and the final bundle works 2. Check :doc:`../topic/repack` .. _target device: @@ -188,6 +203,8 @@ Target Device If your own code in the obfuscated script still isn't executed, check `Bootstrap failed`, otherwise check `failed to run obfuscated scripts` or `failed to run the packed obfuscated scripts` +.. _fix-runtime-bootstrap-issue: + Bootstrap failed ---------------- @@ -203,7 +220,7 @@ Bootstrap failed If it doesn't work, try to use :option:`-i` or :option:`--prefix` to generate the obfuscated scripts again to fix it -3. Try to upgrade Python interprerter to latest patch version. For example, upgrade Python 3.11.0b2 to 3.11.9 +3. Try to upgrade Python interpreter to latest patch version. For example, upgrade Python 3.11.0b2 to 3.11.9 **If target device is different from build device** @@ -244,6 +261,8 @@ If still not work, please check :doc:`environments` to make sure Pyarmor support 2. Use `pyarmor cfg assert.call:excludes "xxx"` and `pyarmor cfg assert.import:excludes "xxx"` to exclude problem modules and functions 3. Find the problem option, and report issue +.. _fix-run-obfuscated-script-issue: + Failed to run obfuscated scripts -------------------------------- @@ -251,7 +270,7 @@ Failed to run obfuscated scripts 2. If it doesn't work, try to obfuscate one simple script, check it works or not -3. Add some print statement in the problem script, and get one script as simple as possible to reproduce the problem. It's better only use Python system packages. If really need third-parth library, check :doc:`../how-to/third-party` first +3. Add some print statement in the problem script, and get one script as simple as possible to reproduce the problem. It's better only use Python system packages. If really need third-party library, check :doc:`../how-to/third-party` first 4. Report issue with necessary information @@ -269,10 +288,12 @@ First use a hello world script to make sure it works. If it doesn't work, check Check solutions in :ref:`using bccmode` +.. _fix-run-packed-script-issue: + Failed to run the packed obfuscated scripts ------------------------------------------- -1. Do not pack the sciprt, just use same options to obfuscate the script, and run the obfuscated script in target device, make sure it works, otherwise check solutions in above section +1. Do not pack the script, just use same options to obfuscate the script, and run the obfuscated script in target device, make sure it works, otherwise check solutions in above section 2. Do not obfuscate the scripts, pack the original script by PyInstaller directly, and execute the final executable in target device, make sure it works. Otherwise check PyInstaller_ documentation to find solutions @@ -291,10 +312,10 @@ Darwin Apple Silicon may need codesign if pyarmor or the obfuscated script can't Download https://pyarmor.dashingsoft.com/downloads/tools/cygcheck.zip and unzip it - Or get it from offical website + Or get it from official website - Open https://cygwin.com/mirrors.html - - Select one mirror, enter pathes `x86_64/release/cygwin/` + - Select one mirror, enter paths `x86_64/release/cygwin/` - Download the latest package `cygwin-3.5.3-1.tar.xz` - Extract `cygcheck.exe` from this file by `tar xJf cygwin-3.5.3-1.tar.xz` diff --git a/docs/topic/bccmode.rst b/docs/topic/bccmode.rst index 6196744d..d073ff33 100644 --- a/docs/topic/bccmode.rst +++ b/docs/topic/bccmode.rst @@ -33,8 +33,8 @@ To check which functions are converted to C function, enable trace mode before o Then check the trace log:: - $ ls .pyarmor/pyarmor.trace.log - $ grep trace.bcc .pyarmor/pyarmor.trace.log + $ ls pyarmor.trace.log + $ grep trace.bcc pyarmor.trace.log trace.bcc foo:5:hello trace.bcc foo:9:sum2 @@ -114,7 +114,7 @@ Let's enable trace mode to check these functions are ignored:: $ pyarmor cfg enable_trace 1 $ pyarmor gen --enable-bcc foo.py - $ grep trace.bcc .pyarmor/pyarmor.trace.log + $ grep trace.bcc pyarmor.trace.log Another example, in the following commands BCC mode ignores ``joker/card.py``, but handle all the other scripts in package ``joker``:: @@ -185,7 +185,31 @@ Here are some changed features in the BCC mode: * Some exception messages may different from the plain script. -* Most of function attributes which starts with ``__`` doesn't exists, or the value is different from the original. +* Most of function attributes which starts with ``__`` doesn't exists, or the value is different from the original. For example, there is no `__qualname__` for BCC function. + +* In the exception handler, `sys.exception()` will return `None`, so the functions depended on `sys.exception` may not work. For example + +.. code-block:: python + + import traceback + + def main(): + try: + 1 / 0 + except Exception as e: + + # In BCC mode, sys.exception() will return None, the output is: + # None + print(sys.exception()) + + # It doesn't work in BCC mode, the output is "NoneType: None" + traceback.print_exc() + + # The traceback will be printed by this form in BCC mode + # But the traceback doesn't include line no. and source + traceback.print_exception(e) + + main() Unsupported features ==================== @@ -216,7 +240,6 @@ And unsupported functions: * super * locals * sys._getframe -* sys.exc_info For example, the following functions are not obfuscated by BCC mode, because they use unsupported features or unsupported functions: @@ -239,6 +262,7 @@ Known issues * When format string has syntax error, BCC mode may raise `SystemError: NULL object passed to Py_BuildValue`, instead of `SyntaxError` or `ValueError`. Found in test cases `lib/python3.12/test/test_fstring.py`: + - test_invalid_syntax_error_message - test_missing_variable - test_syntax_error_for_starred_expressions diff --git a/docs/topic/rftmode.rst b/docs/topic/rftmode.rst index bd5bb4f5..2024208e 100644 --- a/docs/topic/rftmode.rst +++ b/docs/topic/rftmode.rst @@ -56,6 +56,8 @@ There are 2 methods for RFT mode to handle name in the attribute chains which do This method first search all the functions, classes and methods in the scripts, add them to include table, and transform all of them. If same name is used in attribute chains, but can't make sure its type, leave attribute name as it is. + Note that in rft-auto-include mode, local variables will not be touched, but they're renamed in next obfuscation process, unless you explicitly disable it by `pyarmor cfg mix_localnames=0`. + For a simple script, Pyarmor could transform the script automatically. But for a complex script, it may raise name binding error. For example:: $ python dist/foo.py @@ -116,7 +118,7 @@ When both of trace log and trace rft are enabled, RFT mode will log which names $ pyarmor cfg enable_trace=1 trace_rft=1 $ pyarmor gen --enable-rft foo.py - $ grep trace.rft .pyarmor/pyarmor.trace.log + $ grep trace.rft pyarmor.trace.log trace.rft foo:1 (import sys as pyarmor__1) trace.rft foo:12 (self.wScan->self.pyarmor__4) @@ -135,7 +137,7 @@ When RFT scripts complain of name not found error, just exclude this name. For e If no found name like ``pyarmor__22``, find the original name in the trace log:: - $ grep pyarmor__22 .pyarmor/pyarmor.trace.log + $ grep pyarmor__22 pyarmor.trace.log trace.rft foo:65 (self.height->self.pyarmor__22) trace.rft foo:81 (self.height->self.pyarmor__22) @@ -216,7 +218,7 @@ First configure this ruler by command:: Then check the result:: $ pyarmor gen --enable-rft foo.py - $ grep trace.rft .pyarmor/pyarmor.trace.log + $ grep trace.rft pyarmor.trace.log trace.rft foo:8 (self.task.x->self.task.pyarmor__2) @@ -225,7 +227,7 @@ line 8 ``self.task.x`` will be transformed to ``self.task.pyarmor__2`` Let's change action to ``self.?.?``, and check the result:: $ pyarmor cfg rft_rulers "self.task.x self.?.?" - $ grep trace.rft .pyarmor/pyarmor.trace.log + $ grep trace.rft pyarmor.trace.log trace.rft foo:8 (self.task.x->self.pyarmor__1.pyarmor__2) @@ -234,7 +236,7 @@ Do not change action to ``?.?.?``, it doesn't work, the first action can't be `` Let's add new ruler to change ``self.task.y``, here need to use ``^`` to append new line to rulers:: $ pyarmor cfg rft_rulers ^"self.task.y self.?.?" - $ grep trace.rft .pyarmor/pyarmor.trace.log + $ grep trace.rft pyarmor.trace.log trace.rft foo:8 (self.task.x->self.pyarmor__1.pyarmor__2) trace.rft foo:9 (self.task.y->self.pyarmor__1.pyarmor__3) @@ -242,7 +244,7 @@ Let's add new ruler to change ``self.task.y``, here need to use ``^`` to append Actually, both of rulers can combined to one:: $ pyarmor cfg rft_rulers = "self.task.* self.?.?" - $ grep trace.rft .pyarmor/pyarmor.trace.log + $ grep trace.rft pyarmor.trace.log trace.rft foo:8 (self.task.x->self.pyarmor__1.pyarmor__2) trace.rft foo:9 (self.task.y->self.pyarmor__1.pyarmor__3) diff --git a/docs/tutorial/advanced.rst b/docs/tutorial/advanced.rst index 4cf7d6f7..61ec8206 100644 --- a/docs/tutorial/advanced.rst +++ b/docs/tutorial/advanced.rst @@ -143,8 +143,8 @@ To check which functions are converted to C function, enable trace mode before o Then check the trace log:: - $ ls .pyarmor/pyarmor.trace.log - $ grep trace.bcc .pyarmor/pyarmor.trace.log + $ ls pyarmor.trace.log + $ grep trace.bcc pyarmor.trace.log trace.bcc foo:5:hello trace.bcc foo:9:sum2 @@ -252,6 +252,10 @@ Encrypt only string length between 8 and 32 by regular expression:: Check trace log to find which strings are protected. +.. note:: + + This option doesn't touch any docstring + Filter assert function and import ================================= @@ -388,6 +392,8 @@ When obfuscated scripts start, it checks :envvar:`LANG` to get current language export PYARMOR_LANG=zh_TW python dist/foo.py +.. _generating cross platform scripts: + Generating cross platform scripts ================================= @@ -409,6 +415,8 @@ Using :option:`--platform` multiple times to support multiple platforms. For exa --platform darwin.x86_64 \ foo.py +.. _support-multiple-python-versions: + Obfuscating scripts for multiple Python versions ================================================ @@ -476,4 +484,13 @@ If using :term:`outer key` with runtime package, it need specify `--outer` both $ pyarmor gen key -e .10 $ mv dist/pyarmor.rkey dist/pyarmor_runtime_000000 +.. _check-pyarmor-7-license: + +Working with old runtime key +============================ + +If still need check Pyarmor 7 runtime key in the obfuscated scripts of Pyarmor 9, here it's one possible solution + +The idea is still using Pyarmor 7 obfuscated script to verify old runtime key, in Pyarmor 9 obfuscated script check old runtime key by calling Pyarmor 7 obfuscated script indirectly (IPC) + .. include:: ../_common_definitions.txt diff --git a/docs/tutorial/getting-started.rst b/docs/tutorial/getting-started.rst index fc2683ef..d49ba25e 100644 --- a/docs/tutorial/getting-started.rst +++ b/docs/tutorial/getting-started.rst @@ -73,7 +73,7 @@ There is an extra Python package :file:`pyarmor_runtime_000000`, which is requir Distributing the obfuscated script ---------------------------------- -Only copy :file:`dist/foo.py` to another machine doesn't work, instead copy all the files in the :file:`dist/`. +Only copying :file:`dist/foo.py` to another machine will not work. Instead, copy all the files in the :file:`dist/`. Why? It's clear after checking the content of :file:`dist/foo.py`: @@ -82,7 +82,7 @@ Why? It's clear after checking the content of :file:`dist/foo.py`: from pyarmor_runtime_000000 import __pyarmor__ __pyarmor__(__name__, __file__, ...) -Actually the obfuscated script can be taken as normal Python script with dependent package :mod:`pyarmor_runtime_000000`, use it as it's not obfuscated. +The obfuscated script can be taken as a normal Python script with dependent package :mod:`pyarmor_runtime_000000`, use it as it's not obfuscated. .. important:: diff --git a/docs/tutorial/installation.rst b/docs/tutorial/installation.rst index 45b9689a..db21e598 100644 --- a/docs/tutorial/installation.rst +++ b/docs/tutorial/installation.rst @@ -71,6 +71,14 @@ Not all the platforms are supported, more information check :doc:`../reference/e $ pip install pyarmor.cli +.. note:: + + If need install old version Pyarmor, just specify the exact version. For example:: + + $ pip install pyarmor==8.5.12 + + For more information, please check `pip` doc + Installed command ----------------- @@ -170,8 +178,8 @@ Termux issues In Termux, after installation it need patch extensions. For example:: - $ patchelf --add-needed /data/data/com.termux/files/usr/lib/python3.11/site-packages/pyarmor/cli/core/android/aarch64/pytransform3.so - $ patchelf --add-needed /data/data/com.termux/files/usr/lib/python3.11/site-packages/pyarmor/cli/core/android/aarch64/pyarmor_runtime.so + $ patchelf --add-needed libpython3.11.so.0.1 /data/data/com.termux/files/usr/lib/python3.11/site-packages/pyarmor/cli/core/android/aarch64/pytransform3.so + $ patchelf --add-needed libpython3.11.so.0.1 /data/data/com.termux/files/usr/lib/python3.11/site-packages/pyarmor/cli/core/android/aarch64/pyarmor_runtime.so Sometimes, it need set runpath too. For example:: diff --git a/docs/tutorial/obfuscation.rst b/docs/tutorial/obfuscation.rst index 52b98996..7fe0553f 100644 --- a/docs/tutorial/obfuscation.rst +++ b/docs/tutorial/obfuscation.rst @@ -36,18 +36,19 @@ Here only introduces common options in a short, using any combination of them as Debug mode and trace log ======================== -When something is wrong, check console log to find what Pyarmor does, and use :option:`-d` to enable debug mode to print more information:: +When something is wrong, check console log to find what Pyarmor does, and use :option:`-d` to generate :file:`pyarmor.debug.log` to get more information:: $ pyarmor -d gen foo.py + $ cat pyarmor.debug.log Trace log is useful to check whatever protected by Pyarmor, enable it by this command:: $ pyarmor cfg enable_trace=1 -After that, :ref:`pyarmor gen` will generate a logfile :file:`.pyarmor/pyarmor.trace.log`. For example:: +After that, :ref:`pyarmor gen` will generate a logfile :file:`pyarmor.trace.log`. For example:: $ pyarmor gen foo.py - $ cat .pyarmor/pyarmor.trace.log + $ cat pyarmor.trace.log trace.co foo:1: trace.co foo:5:hello @@ -119,7 +120,7 @@ To check which function or which string are protected, enable trace log and chec $ pyarmor cfg enable_trace=1 $ pyarmor gen --mix-str --assert-call fib.py - $ cat .pyarmor/pyarmor.trace.log + $ cat pyarmor.trace.log trace.assert.call fib:10:'fib' trace.mix.str fib:1:'abcxyz' @@ -171,6 +172,42 @@ Now do above test again, it should work:: ... import joker OK ... import joker.queens +.. _using readonly module: + +Using readonly module +--------------------- + +.. versionadded:: 9.1.9 + +Readonly module is one simple way to protect the obfuscated package, it only allows the plain scripts import and read the obfuscated module, but can't write or change any attribute or method of obfuscated modules. + +Enable readonly module by this way:: + + $ pyarmor cfg readonly_module=1 + +Then obfuscate the whole package:: + + $ pyarmor gen --enable-jit --mix-str joker/ + +Test it:: + + $ cd dist + $ python + >>> import joker + >>> dir(joker) + >>> joker.aaa = 1 + Traceback (most recent call last): + File "", line 1, in + RuntimeError: protection exception (16782406) + +If only need export a few modules, for example, `joker.card` and `joker.__init__`, the other modules in the package need not be exported, the best way is like these:: + + $ pyarmor cfg readonly_module=1 + $ pyarmor cfg exclude_restrict_modules="__init__ joker.card" + $ pyarmor gen --enable-jit --mix-str --assert-call --assert-import --restrict joker/ + +The modules list in the `exclude_restrict_modules` are readonly, all the others are more restricted. + Copying package data files ========================== diff --git a/gh.py b/gh.py index 08fd0efe..bf714024 100755 --- a/gh.py +++ b/gh.py @@ -275,6 +275,7 @@ def call_query(cmd, query): def read_body(prompt): + """Input line by line, CTRL+D to end, CTRL+C to cancel""" lines = [] try: @@ -284,6 +285,8 @@ def read_body(prompt): print() lines.append(s) s = input() + except EOFError: + pass except KeyboardInterrupt: lines.clear() print('Abort by user, there is nothing to do') @@ -431,7 +434,7 @@ def do_ad(self, arg): if body: query = query_add_discussion_comment.substitute( discussionId=discussionId, - body=body.replace('"', r'\"')) + body=body.encode("unicode_escape").decode()) call_query(cmd_discussion_graphql, query) def do_dd(self, arg): diff --git a/man/dev.01 b/man/dev.02 similarity index 100% rename from man/dev.01 rename to man/dev.02 diff --git a/plugins/README.md b/plugins/README.md index c0f157bf..52af40c7 100644 --- a/plugins/README.md +++ b/plugins/README.md @@ -1,5 +1,9 @@ # Plugins +***This document is written for Pyarmor 7*** + +***All examples in this folder not work in Pyarmor 8+*** + Plugin usually is used to extend license type, or insert some extra check code to obfuscated scripts to improve the security. diff --git a/build-package.sh b/scripts/build-package.sh similarity index 84% rename from build-package.sh rename to scripts/build-package.sh index 13d51db8..ced4e406 100755 --- a/build-package.sh +++ b/scripts/build-package.sh @@ -2,7 +2,11 @@ # # Build source and wheel distribute # - +# If need new build with same patch version, pass BUILD_NUMBER in command line +# For example, +# +# $ BUILD_NUMBER="--build-number 2" bash scripts/build_package.sh +# PLATFORMS="win32 win_amd64 manylinux1_x86_64 macosx_10_11_x86_64 macosx_10_11_intel" PYTHON=C:/Python34/python @@ -38,16 +42,18 @@ make_platform_files() # cp -a ${src}/macosx_x86_64 ${dst}/darwin/x86_64; cp -a plugins ./src/ } +# Change to workpath +cd ~/workspace/pyarmor # Make platform files make_platform_files # Build source -$PYTHON setup.py sdist --formats=zip,bztar,gztar +$PYTHON setup.py sdist --formats=gztar clear_build # Build universal wheel -$PYTHON setup.py bdist_wheel --python-tag=py3 +$PYTHON setup.py bdist_wheel --python-tag=py3 ${BUILD_NUMBER} clear_build clear_platform_files diff --git a/setup.py b/setup.py index cc1e76aa..dd664e02 100644 --- a/setup.py +++ b/setup.py @@ -113,15 +113,15 @@ }, install_requires=[ - 'pyarmor.cli.core~=6.5.3' + 'pyarmor.cli.core~=8.1.1' ], entry_points={ 'console_scripts': [ - 'pyarmor=pyarmor.pyarmor:main_entry_8', + 'pyarmor=pyarmor.cli.__main__:main', 'pyarmor-auth=pyarmor.cli.docker:main', 'pyarmor-7=pyarmor.pyarmor:main_entry', - 'pyarmor-8=pyarmor.cli.__main__:main', + 'pyarmor-8=pyarmor.pyarmor:main_entry_8', ], }, ) diff --git a/src/README.rst b/src/README.rst index 3336028a..ab156b5e 100644 --- a/src/README.rst +++ b/src/README.rst @@ -6,7 +6,7 @@ Pyarmor is a command line tool used to obfuscate python scripts, bind obfuscated Key Features ------------ -* The obfuscated scritpt is still a normal `.py` script, in most of cases the original python scripts can be replaced with obfuscated scripts seamlessly. +* The obfuscated script is still a normal `.py` script, in most of cases the original python scripts can be replaced with obfuscated scripts seamlessly. * Provide many ways to obfuscate the scripts to balance security and performance * Rename functions/methods/classes/variables/arguments, irreversible obfuscation * Convert part of Python functions to C function, compile to binary by high optimize option, irreversible obfuscation @@ -16,7 +16,7 @@ Key Features Support Platforms ----------------- -* Python 3.7~3.12 +* Python 3.7~3.15 * Windows * Many linuxs, include embedded systems * Apple Intel and Apple Silicon diff --git a/src/cli/__init__.py b/src/cli/__init__.py index 695f3fb4..70ffb8c1 100644 --- a/src/cli/__init__.py +++ b/src/cli/__init__.py @@ -1,6 +1,6 @@ import logging -__VERSION__ = '8.5.12' +__VERSION__ = '9.2.7' logger = logging.getLogger('cli') diff --git a/src/cli/__main__.py b/src/cli/__main__.py index e282432d..9fbeea41 100644 --- a/src/cli/__main__.py +++ b/src/cli/__main__.py @@ -26,13 +26,13 @@ from . import logger, CliError from .context import Context -from .register import Register, WebRegister +from .register import Register, WebRegister, check_license_version from .config import Configer from .shell import PyarmorShell from .plugin import Plugin from .generate import Builder from .bootstrap import check_prebuilt_runtime_library -from .bug import find_solutions +from .command import Commander def _cmd_gen_key(builder, options): @@ -206,7 +206,9 @@ def check_gen_context(ctx, args): if ctx.runtime_outer and any( [ctx.runtime_devices, ctx.runtime_period, ctx.runtime_expired]): - raise CliError('--outer conflicts with any -e, --period, -b') + # Fix issue 2069 + if args.inputs[0].lower() not in ('key', 'k'): + raise CliError('--outer conflicts with any -e, --period, -b') if args.pack: choices = 'onefile', 'onedir', 'F', 'D', 'FC', 'DC' @@ -222,6 +224,8 @@ def check_gen_context(ctx, args): def cmd_gen(ctx, args): options = format_gen_args(ctx, args) logger.debug('command options: %s', options) + check_license_version(ctx) + ctx.push(options) check_gen_context(ctx, args) @@ -305,9 +309,14 @@ def cmd_reg(ctx, args): raise CliError('please use ".zip" file to register group device') regsvr = WebRegister(ctx) regsvr.check_request_interval() - regsvr.register_group_device(regfile, args.device) + regsvr.request_device_regfile(regfile, args.device) logger.info('The device regfile has been generated successfully') + elif args.ci: + regsvr = WebRegister(ctx) + regsvr.check_request_interval() + regsvr.request_ci_regfile(regfile) + elif regfile.endswith('.zip'): reg = Register(ctx) logger.info('register "%s"', regfile) @@ -316,13 +325,13 @@ def cmd_reg(ctx, args): else: regsvr = WebRegister(ctx) - regsvr.check_request_interval() + regsvr.check_request_interval(activation=regfile) info, msg = regsvr.prepare(regfile, args.product, upgrade=upgrade) prompt = 'Are you sure to continue? (yes/no) ' if args.confirm: from time import sleep sleep(1.0) - elif input(msg + prompt) not in ('y', 'yes'): + elif input(msg + prompt) not in ('y', 'yes', 'Y', 'Yes'): logger.info('abort registration') return # Free upgrade to Pyarmor Basic @@ -337,10 +346,33 @@ def cmd_reg(ctx, args): def cmd_man(ctx, args): - logging.info('This feature is still developing ...') + from subprocess import check_call, check_output, STDOUT + + try: + m = __import__('pyarmor.man') + except ModuleNotFoundError: + logger.info('pyarmor.man is still not installed') + if input('Install it now? (Y/n) ') not in ('Y', 'y'): + return + m = None + + if m is None: + try: + logger.info('installing package "pyarmor.man"...') + check_output([sys.executable, '-m', 'pip', + 'install', '-U', 'pyarmor.man'], + stderr=STDOUT) + logger.info('install package "pyarmor.man" OK') + except Exception: + logger.error('install package "pyarmor.man" failed') + logger.error('please install it manually:') + logger.error('\tpip install -U pyarmor.man') + return + + check_call([sys.executable, '-m', 'pyarmor.man.shell']) -def main_parser(): +def main_parser(cmd=None): parser = argparse.ArgumentParser( prog='pyarmor', fromfile_prefix_chars='@', @@ -356,7 +388,7 @@ def main_parser(): ) parser.add_argument( '-d', '--debug', action='store_true', - help='print debug informations in the console' + help='generate debug file "pyarmor.debug.log"' ) parser.add_argument( '-i', dest='interactive', action='store_true', @@ -366,13 +398,17 @@ def main_parser(): subparsers = parser.add_subparsers( title='The most commonly used pyarmor commands are', - metavar='' ) gen_parser(subparsers) reg_parser(subparsers) - cfg_parser(subparsers) + if cmd: + cmd.env_parser(subparsers) + cmd.init_parser(subparsers) + cmd.build_parser(subparsers) + + cfg_parser(subparsers) man_parser(subparsers) return parser @@ -627,6 +663,10 @@ def reg_parser(subparsers): '-y', '--confirm', action='store_true', help=argparse.SUPPRESS ) + cparser.add_argument( + '-C', '--CI', action='store_true', dest='ci', + help='request license regfile for CI pipeline' + ) cparser.add_argument( 'regfile', nargs='?', metavar='FILE', @@ -636,19 +676,18 @@ def reg_parser(subparsers): def man_parser(subparsers): - '''Open Pyarmor.Man in web-browser + '''Start Pyarmor Man shell Pyarmor Man is designed to help Pyarmor users to learn and use Pyarmor, to find solution quickly when something is wrong, to - report bug and ask question by wizard in order to save both - Pyarmor team's and Pyarmor users' time. + report bug by template in order to save both Pyarmor team's + and users' time. ''' - cparser = subparsers.add_parser( 'man', formatter_class=argparse.RawDescriptionHelpFormatter, description=man_parser.__doc__, - help='register Pyarmor or upgrade old Pyarmor license' + help='start Pyarmor Man shell' ) cparser.set_defaults(func=cmd_man) @@ -656,43 +695,45 @@ def man_parser(subparsers): def log_settings(ctx, args): - if args.debug: - root = logging.getLogger() - root.setLevel(logging.DEBUG) - handler = logging.FileHandler(ctx.debug_logfile, - mode='w', - encoding='utf-8') - handler.setFormatter(logging.Formatter('%(asctime)s %(message)s')) - handler.setLevel(logging.DEBUG) - root.addHandler(handler) - - tracelog = logging.getLogger('trace') - tracelog.propagate = False - tracelog.addHandler(logging.NullHandler()) - if ctx.cfg.getboolean('builder', 'enable_trace'): - handler = logging.FileHandler(ctx.trace_logfile, - mode='w', - encoding='utf-8') - handler.setFormatter(logging.Formatter('%(name)-20s %(message)s')) - handler.setLevel(logging.DEBUG if args.debug else logging.INFO) - tracelog.addHandler(handler) + import logging.config + logging.config.fileConfig(ctx.default_config) if args.silent: - logging.getLogger().setLevel(100) + logger.setLevel(logging.ERROR) + + elif args.debug: + logger.setLevel(logging.DEBUG) + logger.handlers[1].setLevel(logging.DEBUG) + + if ctx.cfg.getboolean('builder', 'enable_trace'): + level = logging.DEBUG if args.debug else logging.INFO + logging.getLogger('trace').setLevel(level) + + +def log_bug(e): + parser = main_parser(cmd=Commander()) + args = parser.parse_args(sys.argv[1:]) + ctx = Context(*get_home_paths(args)) + cmdline = ' '.join(sys.argv) + buglog = logging.getLogger('cli.bug') + buglog.info('[BUG]: %s\n', e) + buglog.info('## Command Line\n%s\n', cmdline) + buglog.info('## Environments', ) + buglog.info('Python %d.%d.%d', *sys.version_info[:3]) + buglog.info('Pyarmor %s', ctx.version_info()) + buglog.info('Platform %s', ctx.pyarmor_platform) + buglog.info('Native %s', ctx.native_platform) + buglog.info('Home %s', ctx.home_path) + buglog.info('') -def log_exception(e): - logger.debug('unknown error, please check pyarmor.error.log') - handler = logging.FileHandler('pyarmor.error.log', - mode='w', - encoding='utf-8') - fmt = '%(process)d %(processName)s %(asctime)s' - handler.setFormatter(logging.Formatter(fmt)) - log = logging.getLogger('error') - log.propagate = False - log.addHandler(logging.NullHandler()) - log.addHandler(handler) - log.exception(e) + if not isinstance(e, CliError): + from traceback import format_exc + buglog.info('## Traceback\n%s\n', format_exc()) + + # reference/errors.html + # reference/solutions.html + # questions.html def print_version(ctx): @@ -701,6 +742,7 @@ def print_version(ctx): print('\n'.join(info)) reg.check_group_license() + check_license_version(ctx, silent=True) def get_home_paths(args): @@ -714,15 +756,19 @@ def get_home_paths(args): def main_entry(argv): - parser = main_parser() + cmd = Commander() + parser = main_parser(cmd) args = parser.parse_args(argv) - if sys.version_info[0] == 2 or sys.version_info[1] < 7: - raise CliError('only Python 3.7+ is supported now') + ctx = cmd.ctx = Context(*get_home_paths(args)) + log_settings(ctx, args) - ctx = Context(*get_home_paths(args)) + x, y = sys.version_info[:2] + if not (x == 3 and y > 6 and y < 16): + raise CliError('Python %s.%s is not supported' % (x, y)) - log_settings(ctx, args) + if ctx.py_gil_disabled: + raise CliError('Pyarmor does not support free-threading Python') if args.version: print_version(ctx) @@ -737,6 +783,7 @@ def main_entry(argv): logger.debug('native platform %s', ctx.native_platform) logger.debug('home path: %s', ctx.home_path) + logger.debug('args: %s', argv) if hasattr(args, 'func'): return args.func(ctx, args) @@ -744,23 +791,21 @@ def main_entry(argv): parser.print_help() -def main(): - logging.basicConfig( - level=logging.INFO, - format='%(levelname)-8s %(message)s', - ) +def map_error(e): + if str(e) == 'invalid license token': + return ('invalid license token or failed to update license token. ' + 'please make sure this device is online ' + 'and can connect to Pyarmor License Server') + return e + +def main(): try: main_entry(sys.argv[1:]) - except CliError as e: - logger.error(e) - find_solutions(e) - sys.exit(1) except Exception as e: - log_exception(e) - logger.error(e) - find_solutions(e) - sys.exit(2) + logger.error(map_error(e)) + log_bug(e) + sys.exit(1 if isinstance(e, CliError) else 2) if __name__ == '__main__': diff --git a/src/cli/bootstrap.py b/src/cli/bootstrap.py index af961433..10ef540a 100644 --- a/src/cli/bootstrap.py +++ b/src/cli/bootstrap.py @@ -47,7 +47,7 @@ def check_prebuilt_runtime_library(platnames, extra=None, rtver=''): if current_rtver == rtver: return - pkgver = 'pyarmor.cli.runtime==%s' % rtver + pkgver = 'pyarmor.cli.runtime~=%s' % rtver logging.info('install "%s" for cross platforms', pkgver) try: return check_call(instcmd + [pkgver]) @@ -73,7 +73,7 @@ def check_prebuilt_runtime_library(platnames, extra=None, rtver=''): pkgnames.remove(entry.name) if pkgnames: - pkgvers = ['pyarmor.cli.core.%s==%s' % (x, corever) for x in pkgnames] + pkgvers = ['pyarmor.cli.core.%s~=%s' % (x, corever) for x in pkgnames] logging.info('install packages %s for cross platforms', str(pkgvers)) try: check_call(instcmd + pkgvers) diff --git a/src/cli/bug.py b/src/cli/bug.py index 02e03715..69c5a122 100644 --- a/src/cli/bug.py +++ b/src/cli/bug.py @@ -18,7 +18,14 @@ # @Author: Jondy Zhao (pyarmor@163.com) # # @Create Date: Tue Mar 12 15:19:41 CST 2024 -# + + +############################################################# +# # +# Deprecated since Pyarmor 9, use `pyarmor man` instead # +# # +############################################################# + import logging from string import Template @@ -62,7 +69,7 @@ ENABLE_DEBUG_HINTS = 'please enable debug option `-d` to run it again' -SOLUTION_HINTS = Template('''something is wrong +SOLUTION_HINTS = Template(''' *=============================================================* * Please check console log to find out what's wrong * * * @@ -118,12 +125,12 @@ def find_solutions(e): ''' logger = logging.getLogger() if logger.getEffectiveLevel() > logging.DEBUG: - logger.error(ENABLE_DEBUG_HINTS) return + print('\nDebug Information:') logger.info('generate bug file "%s"', BUG_FILE) generate_bug_report(e) - logger.error(SOLUTION_HINTS.substitute(bugfile=BUG_FILE)) + logger.info(SOLUTION_HINTS.substitute(bugfile=BUG_FILE)) if __name__ == '__main__': diff --git a/src/cli/command.py b/src/cli/command.py new file mode 100644 index 00000000..4bd7d197 --- /dev/null +++ b/src/cli/command.py @@ -0,0 +1,563 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- +# +############################################################# +# # +# Copyright @ 2024 Dashingsoft corp. # +# All rights reserved. # +# # +# pyarmor # +# # +# Version: 9.1.0 - # +# # +############################################################# +# +# +# @File: pyarmor/cli/commander.py +# +# @Author: Jondy Zhao(pyarmor@163.com) +# +# @Create Date: Tue Nov 12 16:38:51 CST 2024 +# +import argparse +import configparser +import shlex + +from os import makedirs, remove as rmfile +from os.path import abspath, exists, join as joinpath, relpath + +from . import logger, CliError +from .project import Project +from .shell import PyarmorShell + + +class Commander: + + def init_parser(self, subparsers): + parser = subparsers.add_parser( + 'init', + aliases=['i'], + formatter_class=argparse.RawDescriptionHelpFormatter, + description='init project in current path', + help='init project in current path' + ) + + parser.add_argument( + '-s', '--src', metavar='PATH', default='', + help='where to find scripts, modules and packages' + ) + parser.add_argument( + '-e', '--entry', dest='scripts', metavar='FILE', + action='append', + help='project entry scripts' + ) + parser.add_argument( + '-m', '--module', metavar='FILE', + dest='modules', action='append', + help='append extra module' + ) + parser.add_argument( + '-p', '--package', metavar='PATH', + dest='packages', action='append', + help='append extra package' + ) + + parser.add_argument( + '-x', '--exclude', metavar='PATTERN', + dest='excludes', action='append', + help='exclude file or path' + ) + + parser.add_argument( + '-r', '--recursive', action='store_true', + help='Search modules and packages recursively' + ) + parser.add_argument( + '-C', '--clean', action='store_true', + help='remove old project information before init' + ) + + parser.set_defaults(func=self.cmd_init) + + def env_parser(self, subparsers): + parser = subparsers.add_parser( + 'env', + aliases=['environ', 'e'], + formatter_class=argparse.RawDescriptionHelpFormatter, + description='check and set Pyarmor environments', + help='check and set Pyarmor environments' + ) + + # parser.add_argument( + # '-i', '--interactive', action='store_true', + # help='enable interactive mode' + # ) + + group = parser.add_argument_group( + 'select domain (default: project)' + ).add_mutually_exclusive_group() + group.add_argument( + '-l', '--local', dest='domain', default='local', + action="store_const", const='local', + help='enter local domain' + ) + group.add_argument( + '-g', '--global', dest='domain', default='local', + action="store_const", const='global', + help='enter global domain' + ) + group.add_argument( + '-p', '--project', dest='domain', default='local', + action="store_const", const='project', + help='enter project domain' + ) + + parser.add_argument( + 'exprs', metavar='VERB', nargs='*', + help='change option by info/ls/set/reset/push/pop' + ) + + parser.set_defaults(func=self.cmd_env) + + def build_parser(self, subparsers): + parser = subparsers.add_parser( + 'build', + aliases=['b'], + formatter_class=argparse.RawTextHelpFormatter, + help='obfuscate all the scripts in the project' + ) + + group = parser.add_argument_group( + 'select build target' + ).add_mutually_exclusive_group() + # group.add_argument( + # '--std', dest='target', default='std', + # action="store_const", const='std', + # help='genetate standard obfuscated scripts' + # ) + group.add_argument( + '--rft', dest='target', default='std', + action="store_const", const='rft', + help='only refactor scripts' + ) + group.add_argument( + '--mini', dest='target', default='std', + action="store_const", const='mini', + help='genetate high performance scripts' + ) + group.add_argument( + '--mini-rft', dest='target', default='std', + action="store_const", const='mini-rft', + help='generate scripts by combining --rft and --mini' + ) + group.add_argument( + '--vmc', dest='target', default='std', + action="store_const", const='vmc', + help='genetate scripts with vmc code' + ) + group.add_argument( + '--vmc-rft', dest='target', default='std', + action="store_const", const='vmc-rft', + help='generate scripts by combining --rft and --vmc' + ) + group.add_argument( + '--ecc', dest='target', default='std', + action="store_const", const='ecc', + help='genetate scripts with embedded C code' + ) + group.add_argument( + '--ecc-rft', dest='target', default='std', + action="store_const", const='ecc-rft', + help='generate scripts by combining --rft and --ecc' + ) + group.add_argument( + '--ecc-nogil', dest='target', default='std', + action="store_const", const='ecc-nogil', + help='genetate ECC scripts for free-threading Python' + ) + group.add_argument( + '--ecc-rft-nogil', dest='target', default='std', + action="store_const", const='ecc-rft-nogil', + help=('generate scripts by combining --rft and --ecc' + ' for free-threading Python') + ) + group.add_argument( + '--list', dest='target', default='std', + action="store_const", const='list', + help='list project scripts, modules and packages' + ) + group.add_argument( + '--types', dest='target', default='std', + action="store_const", const='types', + help=argparse.SUPPRESS + ) + group.add_argument( + '--randname', type=int, metavar='{0,1}', + help='Build random name pool' + ) + group.add_argument( + '--autofix', type=int, choices=(0, 1, 2, 3), + help='Generate refactor rules by autofix mode' + ) + + # parser.add_argument( + # '--pack', metavar='MODE', + # help='specify pack mode, onefile or onedir' + # ) + + # sgroup = parser.add_argument_group( + # title='these options only for --std target', + # ) + # sgroup.add_argument( + # '--restrict', + # default=1, choices=(0, 1, 2), + # help='set restrict mode (default: 1)' + # ) + # sgroup.add_argument( + # '--assert-import', action="store_true", + # help='check imported module is obfuscated' + # ) + # sgroup.add_argument( + # '--assert-call', action="store_true", + # help='check function is obfuscated before call' + # ) + + parser.add_argument( + '-O', '--output', metavar='PATH', + help='output path, default is "dist"' + ) + + # parser.add_argument( + # 'project', metavar='PATH', nargs='?', + # help='project path (default: current path)' + # ) + + parser.set_defaults(func=self.cmd_build) + + def main_parser(self): + parser = argparse.ArgumentParser( + prog='pyarmor', + fromfile_prefix_chars='@', + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + '-v', '--version', action='store_true', + help='show version information and exit' + ) + parser.add_argument( + '-q', '--silent', action='store_true', + help='suppress all normal output' + ) + + subparsers = parser.add_subparsers( + title='The most commonly used pyarmor commands are', + ) + + self.env_parser(subparsers) + self.init_parser(subparsers) + self.build_parser(subparsers) + + return parser + + def run(self, ctx, argv): + parser = self.main_parser() + args = parser.parse_args(argv) + if hasattr(args, 'func'): + args.func(ctx, args) + else: + parser.print_help() + + def cmd_init(self, ctx, args): + logger.debug('init %s', args) + + sep = ',' + cfgsep = ' ' + data = {} + + cfg = configparser.ConfigParser( + empty_lines_in_values=False, + interpolation=configparser.ExtendedInterpolation(), + ) + cfg.read([ctx.local_config], encoding=ctx.encoding) + + sectname = 'project' + if args.clean: + logger.info('clean old project') + if cfg.has_section(sectname): + logger.debug('remove cfg section: project') + cfg.remove_section(sectname) + if cfg.has_section('rft'): + logger.debug('remove cfg section: rft') + cfg.remove_section('rft') + propath = joinpath(ctx.local_path, 'project') + for name in ['rft_autofix.rules']: + profile = joinpath(propath, name) + if exists(profile): + logger.debug('remove file: %s', profile) + rmfile(profile) + + if cfg.has_section(sectname): + logger.info('change project settings') + else: + cfg.add_section(sectname) + + sect = cfg[sectname] + if args.src: + src = abspath(args.src) + if not sect.get('src'): + sect['src'] = src + elif src != sect['src']: + raise CliError( + 'project has another src, ' + 'use option "-C" to fix this issue' + ) + else: + src = sect.get('src') + if not src: + src = abspath('.') + sect['src'] = src + + if not exists(src): + raise CliError('no found src "%s"' % src) + + def format_path(plist, raw=False): + r = [] + for x in sep.join(plist).split(sep): + x = x.strip() + r.append(x if raw else relpath(abspath(x), src)) + return r + + if args.scripts: + data['scripts'] = format_path(args.scripts) + + if args.excludes: + data['excludes'] = format_path(args.excludes, raw=1) + + if args.modules: + data['modules'] = format_path(args.modules) + + if args.packages: + data['packages'] = format_path(args.packages) + + if args.recursive: + data['recursive'] = '1' + + for key, value in data.items(): + old = sect.get(key) + if old is not None: + logger.info('overwrite %s: %s', key, old) + if isinstance(value, (tuple, list)): + value = cfgsep.join([ + x.replace(' ', '%20%') for x in value + ]) + sect[key] = value + + logger.info('project information') + for key, value in cfg.items(sectname): + if key in ('src',): + value = relpath(value) + logger.info('%-20s: %s', key, value) + + makedirs(ctx.local_path, exist_ok=True) + with open(ctx.local_config, 'w', encoding=ctx.encoding) as f: + cfg.write(f) + logger.info('project saved') + + def cmd_build(self, ctx, args): + """Obfuscate all scripts, modules and packages in project + + Show project information: + + pyarmor i + + Generate plain scripts which no extra extension, the final + scripts could be taken as input for Nuitka, Cython etc.: + + pyarmor build --rft + + Generate high performance scripts: + + pyarmor build --mini + + """ + logger.debug('build %s', args) + cfg = configparser.ConfigParser( + empty_lines_in_values=False, + interpolation=configparser.ExtendedInterpolation(), + ) + cfg.read([ctx.local_config], encoding=ctx.encoding) + + x, y = ctx.python_version + if not (x == 3 and y > 8 and y < 16): + raise CliError('`pyarmor build` only works for Python 3.9+') + + sectname = 'project' + if not cfg.has_section(sectname): + raise CliError('no project information') + + cfgdata = dict(cfg.items(sectname)) + logger.debug('project: %s', cfgdata) + + project = Project(ctx) + logger.info('load project ...') + project.load(cfgdata) + logger.info('load project end') + + if args.target == 'list': + logger.info('project src: %s', relpath(project.src)) + logger.info('project scripts:') + for x in project.scripts: + logger.info(' %s', project.relsrc(x.abspath)) + logger.info('project modules:') + for x in project.modules: + logger.info(' %s', project.relsrc(x.abspath)) + logger.info('project packages:') + for x in project.packages: + logger.info(' %s', x.name) + mlist = [m.abspath for m in project.iter_module()] + mlist.sort() + for s in mlist: + logger.info(' %s', project.relsrc(s)) + return + + if args.autofix is not None: + value = args.autofix + logger.info('build auto-fix-table:%s ...', value) + output = args.output if args.output else 'dist' + self._build(project, 'autofix', output, value) + logger.info('build auto-fix-table:%s end', value) + project.preview_autofix_result(value) + elif args.randname is not None: + value = args.randname + logger.info('build rand-pool:%s ...', value) + output = args.output if args.output else 'dist' + self._build(project, 'namepool', output, value) + logger.info('build rand-pool:%s end', value) + else: + logger.info('build target %s ...', args.target) + output = args.output if args.output else 'dist' + self._build(project, args.target, output) + if args.target.startswith('std'): + self._cmd_gen(args) + logger.info('build target %s end', args.target) + + def _build(self, project, target, output, value=None): + from pyarmor.cli.core import Pytransform3 + args = [self.ctx, target, project, output, value] + m = Pytransform3.init(self.ctx) + m.pre_build(args) + + def _cmd_gen(self, args): + """If build target is std, then call `pyarmor gen` + + It will obfuscate the scripts with `pyarmor_runtime.so` + + Most of obfuscated options are got from config file, not from + command line + + Option `--pack` may be still got from command line + """ + from .generate import Builder + from .plugin import Plugin + + ctx = self.ctx + builder = Builder(ctx) + Plugin.install(ctx) + builder.process(args) + Plugin.post_build(ctx) + + def cmd_env(self, ctx, args): + """Check and change pyarmor settings + + Enter interactive mode: + + pyarmor env + + Show project src + + pyarmor env -p get src + + Change project src + + pyarmor env -p set src ../src + + Append project excludes + + pyarmor env -p push excludes "test*" + + Remove project excludes + + pyarmor env -p pop excludes "test*" + + Clear project excludes + + pyarmor env -p reset excludes + + """ + logger.debug('env %s', args) + logger.info('enter domain: %s', args.domain) + shell = PyarmorShell(ctx, domain=args.domain) + + if args.exprs: + exprs = args.exprs + n = len(exprs) + if n < 2: + logger.error('missing parameters') + return + + verb = exprs[0] + opt = exprs[1] + + i = opt.find(':') + if i > 0: + section, opt = opt[:i], opt[i+1:] + shell.do_cd(section) + + if verb == 'reset': + shell.do_reset(opt) + elif verb == 'get': + shell.do_get(opt) + elif verb == 'info': + shell.do_info(opt) + elif n < 3: + logger.error('missing parameters') + return + else: + arg = shlex.join([opt] + exprs[2:]) + if verb == 'pop': + shell.do_pop(arg) + elif verb == 'push': + shell.do_push(arg) + elif verb == 'set': + shell.do_set(arg) + else: + logger.error('unknown verb "%s"', verb) + return + + shell.save() + + else: + shell.cmdloop() + + +def test_main(args, target='rft', log=False): + import logging.config + from os.path import expanduser + + from .context import Context + from .rftbuild import rft_build_project + + home = joinpath('~', '.pyarmor') + home = abspath(expanduser(home)) + + ctx = Context(home) + if log: + logging.config.fileConfig(ctx.default_config) + cmd = Commander() + cmd._build = rft_build_project + cmd.run(ctx, args) + + +if __name__ == '__main__': + import sys + test_main(sys.argv[1:], log=True) diff --git a/src/cli/config.py b/src/cli/config.py index 1829ccb0..3c8f2a9a 100644 --- a/src/cli/config.py +++ b/src/cli/config.py @@ -43,7 +43,7 @@ class Configer(object): def __init__(self, ctx, encoding=None): self.ctx = ctx - self._encoding = encoding + self._encoding = encoding or ctx.encoding def _read_config(self, filename): cfg = configparser.ConfigParser(empty_lines_in_values=False) @@ -148,7 +148,7 @@ def _set_option(self, sect, opt, value, local=True, name=None): if not optvalue: if op is None: - self._clear(sect, optname, local, name) + self._remove(sect, [optname], local, name) return ctxcfg = ctx.cfg @@ -166,7 +166,7 @@ def _set_option(self, sect, opt, value, local=True, name=None): cfg.set(sect, optname, optvalue) os.makedirs(os.path.dirname(filename), exist_ok=True) - with open(filename, 'w') as f: + with open(filename, 'w', encoding=self._encoding) as f: cfg.write(f) self._list_value(sect, optname, local=local, name=name) @@ -196,7 +196,7 @@ def _remove(self, section=None, options=None, local=True, name=None): logger.info('remove empty section "%s"', section) cfg.remove_section(section) - with open(filename, 'w') as f: + with open(filename, 'w', encoding=self._encoding) as f: cfg.write(f) def _clear(self, section=None, options=None, local=True, name=None): diff --git a/src/cli/context.py b/src/cli/context.py index 068ad359..a3c73969 100644 --- a/src/cli/context.py +++ b/src/cli/context.py @@ -164,9 +164,9 @@ def __init__(self, home, gpath='', lpath='', rpath='', encoding=None): os.path.join(self.home_path, rpath) # self.encoding is just for reading config file - self.encoding = encoding + self.encoding = encoding if encoding else 'utf-8' cfglist = self.default_config, self.global_config, self.local_config - self.cfg = self._read_config(cfglist, encoding=encoding) + self.cfg = self._read_config(cfglist, encoding=self.encoding) # self.runtime_package = 'pyarmor_runtime' # self.runtime_suffix = '_000000' @@ -205,6 +205,7 @@ def _read_config(self, filelist, encoding=None): empty_lines_in_values=False, interpolation=configparser.ExtendedInterpolation(), ) + encoding = encoding or self.encoding cfg.read(filelist, encoding=encoding) return cfg @@ -301,9 +302,10 @@ def version_info(self, verbose=3): return rev licinfo = self.license_info - lictype = 'basic' if licinfo['features'] == 1 else \ + lictype = 'basic' if licinfo['features'] in (1, 17) else \ 'pro' if licinfo['features'] == 7 else \ 'group' if licinfo['features'] == 15 else \ + 'ci' if licinfo['features'] == 23 else \ 'trial' if licinfo['token'] == 0 else 'unknown' verinfo = ['%s (%s)' % (rev, lictype)] @@ -329,6 +331,11 @@ def version(self): def python_version(self): return sys.version_info[:2] + @property + def py_gil_disabled(self): + from sysconfig import get_config_var + return 1 if get_config_var("Py_GIL_DISABLED") else 0 + @property def default_config(self): return os.path.join(os.path.dirname(__file__), 'default.cfg') diff --git a/src/cli/core/__init__.py b/src/cli/core/__init__.py index 8c42d2fa..8622ee2a 100644 --- a/src/cli/core/__init__.py +++ b/src/cli/core/__init__.py @@ -20,7 +20,7 @@ # @Create Date: Thu Jan 12 17:29:25 CST 2023 # -__VERSION__ = '6.5.3' +__VERSION__ = '8.1.1' PLATFORM_NAMES = ( 'windows.x86_64', 'windows.x86', 'cygwin.x86_64', @@ -50,7 +50,7 @@ def check_and_install_prebuilt_package(): platname = os.getenv( 'PYARMOR_PLATFORM', - '.'.join(format_platform(system().lower(), machine().lower()))) + format_platform(system().lower(), machine().lower())) platname = map_platform(platname) if platname not in PLATFORM_NAMES: raise RuntimeError('"%s" is still not supported by Pyarmor' % platname) diff --git a/src/cli/default.cfg b/src/cli/default.cfg index 9dba93c7..41b772e6 100644 --- a/src/cli/default.cfg +++ b/src/cli/default.cfg @@ -1,12 +1,16 @@ [pyarmor] ;; Pyarmor version -major = 8 -minor = 5 -patch = 12 +major = 9 +minor = 2 +patch = 7 -;; Compatible core version -cli.core = 6.5.3 +;; Compatible core version for package: pyarmor.cli.core +cli.core = 8.1.1 + +;; Compatible core version for package: pyarmor.mini +;; Which is required to run MINI, VMC, or ECC mode scripts +mini.core = 3.0 ;; Deprecated since Pyarmor 8.2.5 ; cli.runtime = 3.2.5 @@ -17,12 +21,8 @@ cli.core = 6.5.3 timeout = 6 regurl = https://api.dashingsoft.com/product/key/enter/%s/? -buyurl = https://order.mycommerce.com/product?vendorid=200089125&productid=301044051 - -[logging] - -debug_logfile = .pyarmor/pyarmor.debug.log -trace_logfile = .pyarmor/pyarmor.trace.log +buyurl = https://jondy.github.io/paypal/index.html +docurl = https://pyarmor.readthedocs.io/{lang}/latest [finder] recursive = 0 @@ -148,6 +148,16 @@ enable_jit = 0 enable_bcc = 0 enable_rft = 0 +;; Not implemented +;; Enable opcode virtual protection +;; This feature is meaningless in BCC mode +enable_vmc = 0 + +;; New in v9.2.0 +;; Enable to obfuscate attribute name +;; When obf_code == 2, it's enabled implicitly +mix_attr = 0 + ;; assert: call import assert_call = 0 assert_import = 0 @@ -166,7 +176,16 @@ mix_localnames = 1 ;; mix argument names, it also clears annotations mix_argnames = 0 +;; Obfuscate the whole module obf_module = 1 + +;; Obfuscate each code object +;; +;; 0: no obfuscation for each co +;; 1: obfuscate each co +;; 2: obfuscate each co plus mix attribute name +;; this is deprecated since 9.2.0, use +;; obf_code == 1 and mix_attr = 1 obf_code = 1 ;; 0: no wrap mode @@ -175,8 +194,15 @@ obf_code = 1 ;; Since Python 3.12, "2" is same as "1", both of them use full wrap mode wrap_mode = 1 +;; 1: the obfuscated scripts can't be modified +;; 2: private module, can't be imported by plain scripts +;; 3: restrict module, module attribute is hidden for outer scripts +;; restrict_module = 1 +;; if set to 1, module attribute can't be written by plain scripts +readonly_module = 0 + ;; ;; Advanced features ;; @@ -209,6 +235,13 @@ rft_mix_import_name = 0 ;; Strip leading name package strip_package_name = 1 +;; Enable ProcessPoolExecuter to generate the obfuscated scripts +;; +;; 0: normal build +;; 1: use ProcessPoolExecutor with max_workers=None +;; N > 1: use ProcessPoolExecutor with max_workers=N +max_workers = 0 + ;; ;; Compatiable options ;; @@ -455,3 +488,77 @@ cflags = -O3 -Wno-unsequenced -fno-asynchronous-unwind-tables -fno-unwind-tables [freebsd.x86_64.bcc] cflags = -O3 -Wno-unsequenced -fno-asynchronous-unwind-tables -fno-unwind-tables -fPIC -fno-stack-protector -c + +;;;;;;;;;;;;;;;;;;;;;; +;; +;; Logging settings +;; +;;;;;;;;;;;;;;;;;;;;;; +[loggers] +keys=root,cli,trace,bug + +[handlers] +keys=console,trace,debug,bug + +[formatters] +keys=console,debug,trace,bug + +[logger_root] +handlers=console + +[logger_cli] +level=INFO +handlers=console,debug +qualname=cli +propagate=0 + +[logger_trace] +level=ERROR +handlers=trace +qualname=trace +propagate=0 + +[logger_bug] +level=INFO +handlers=bug +qualname=cli.bug +propagate=0 + +[handler_console] +class=StreamHandler +level=INFO +formatter=console +args=(sys.stderr,) + +[handler_debug] +class=FileHandler +level=CRITICAL +formatter=debug +args=('pyarmor.debug.log', 'w') +kwargs={'delay': True} + +[handler_trace] +class=FileHandler +level=DEBUG +formatter=trace +args=('pyarmor.trace.log', 'w') +kwargs={'delay': True} + +[handler_bug] +class=FileHandler +level=INFO +formatter=bug +args=('pyarmor.bug.log', 'w') +kwargs={'delay': True} + +[formatter_console] +format=%(levelname)-8s %(message)s + +[formatter_debug] +format=%(asctime)s %(message)s + +[formatter_trace] +format=%(name)-20s %(message)s + +[formatter_bug] +format=%(message)s diff --git a/src/cli/generate.py b/src/cli/generate.py index 1d6ec65c..240a113b 100644 --- a/src/cli/generate.py +++ b/src/cli/generate.py @@ -22,8 +22,9 @@ import os import shutil +from pyarmor.cli.core import Pytransform3 + from . import logger, CliError -from .core import Pytransform3 from .resource import FileResource, PathResource @@ -185,7 +186,71 @@ def process(self, options, packer=None): logger.info('generate runtime files OK') logger.info('start to obfuscate scripts') - self._obfuscate_scripts() + n = self.ctx.cfg.getint('builder', 'max_workers') + async_obfuscate_scripts(self, n) if n else self._obfuscate_scripts() logger.info('obfuscate scripts OK') Pytransform3.post_build(self.ctx) + + +def async_obfuscate_scripts(builder, max_workers): + from concurrent.futures import ProcessPoolExecutor, wait + + ctx = builder.ctx + rev = ctx.version_info() + template = ctx.bootstrap_template + relative = ctx.import_prefix + pkgname = ctx.runtime_package_name + bootpath = ctx.cfg.get('builder', 'bootstrap_file') + encoding = ctx.cfg.get('builder', 'encoding') + max_workers = None if max_workers < 2 else max_workers + + plugins = [x for x in ctx.plugins if hasattr(x, 'post_script')] + + namelist = [] + futures = [] + + with ProcessPoolExecutor(max_workers=max_workers) as executor: + for res in ctx.resources + ctx.extra_resources: + logger.info('process resource "%s"', res.fullname) + name = res.name + path = builder.format_output(ctx.outputs, namelist.count(name)) + namelist.append(name) + os.makedirs(path, exist_ok=True) + + for r in res: + if not r.is_script(): + logger.info('copy data file %s', r.fullpath) + data_path = os.path.join(path, r.output_path) + os.makedirs(data_path, exist_ok=True) + shutil.copy2(r.fullpath, data_path) + continue + + logger.info('obfuscating %s', r) + futures.append(executor.submit(aysnc_generate_script, *( + r, path, ctx, template, relative, pkgname, bootpath, + rev, plugins, encoding))) + + wait(futures) + for x in futures: + x.result() + + +def aysnc_generate_script(r, path, ctx, template, relative, + pkgname, bootpath, rev, plugins, encoding): + code = Pytransform3.generate_obfuscated_script(ctx, r) + source = r.generate_output( + template, code, relative=relative, pkgname=pkgname, + bootpath=bootpath, rev=rev + ) + + fullpath = os.path.join(path, r.output_filename) + os.makedirs(os.path.dirname(fullpath), exist_ok=True) + + for plugin in plugins: + patched_source = plugin.post_script(ctx, r, source) + if patched_source: + source = patched_source + + with open(fullpath, 'w', encoding=encoding) as f: + f.write(source) diff --git a/src/cli/model.py b/src/cli/model.py new file mode 100644 index 00000000..697d8093 --- /dev/null +++ b/src/cli/model.py @@ -0,0 +1,460 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- +# +############################################################# +# # +# Copyright @ 2024 - Dashingsoft corp. # +# All rights reserved. # +# # +# Pyarmor # +# # +# Version: 9.1.0 - # +# # +############################################################# +# +# +# @File: cli/model.py +# +# @Author: Jondy Zhao (pyarmor@163.com) +# +# @Create Date: Sun Dec 8 07:36:16 CST 2024 +# +from textwrap import dedent + + +class Item: + + def __init__(self, name, cls=None, require=False, many=False): + self.name = name + self.cls = cls + self.require = require + self.many = many + + +class Option: + + def __init__(self, name, parent=None, model=None): + self.name = name + self.parent = parent + self.model = model + self._value = None + + @property + def value(self): + if self._value is None: + self._value = self.parent.fetch(self.name) + if self._value is None: + return self.model.value + return self._value + + @value.setter + def value(self, data: str): + if data: + self.check(data) + self._value = data + self.parent.store(self.name, data) + else: + self.reset() + + def check(self, data: str): + pass + + def hints(self): + return '' + + def usage(self): + fmt = '%-20s: %s' + hints = [self.hints()] + hints.extend([(fmt % x) for x in self.model.hints()]) + if getattr(self, 'CHOICES', None): + values = ', '.join(self.CHOICES) + hints.append(fmt % ('Available values', values)) + hints.append('') + return '\n'.join(hints) + + def push(self, data: str): + items = self.value.splitlines() if self.value else [] + value = data.strip() + if value not in items: + items.append(value) + self.value = '\n'.join(items) + + def pop(self, data: str): + items = self.value.splitlines() if self.value else [] + try: + items.remove(data.strip()) + except ValueError: + pass + else: + self.value = '\n'.join(items) + + def reset(self): + """Clear option settings""" + self.parent.remove(self.name) + self._value = None + + +class OptionModel: + + def __init__(self, name, cls=None, **kwargs): + self.name = name + self.cls = cls + self.kwargs = kwargs + + def factory(self, parent): + Cls = globals().get(self.cls, Option) + return Cls(self.name, parent=parent, model=self) + + @property + def many(self): + """This option may has many values""" + return self.kwargs.get('many', False) + + @property + def value(self): + """The default value for this option""" + return self.kwargs.get('value') + + def hints(self): + return [('Type', self.cls), ('Many', self.many)] + + +class PathOption(Option): + pass + + +class PkgPathOption(PathOption): + + def hints(self): + return dedent("""\ + It could be relative path or absolute path. + + If package name is different from last path, use suffix format "path@pkgname" + """) + + +class FileOption(PathOption): + pass + + +class BoolOption(Option): + + CHOICES = '0', '1' + + def check(self, data: str): + if data not in self.CHOICES: + raise ValueError('invalid %s' % repr(data)) + + +class EnumOption(Option): + + def check(self, data: str): + if data not in self.CHOICES: + raise ValueError('invalid %s' % repr(data)) + + +class TextOption(Option): + pass + + +class NameOption(TextOption): + pass + + +class PatternOption(TextOption): + """re or fnmatch pattern""" + pass + + +class ListOption(Option): + pass + + +class DictOption(Option): + pass + + +class RftAttrEnum(EnumOption): + """When don't know how to rename attribute + + - ask: query user interactively + - log: log it but no rename + - yes: always rename + - no: do thing + - err: raise error quit + """ + + CHOICES = 'ask', 'log', 'yes', 'no', 'err' + + +class RftArgEnum(EnumOption): + """How to rename argument + + - 0: no rename arguments + - 1: rename posonly arguments + - 2: rename kwonly arguments + - 3: rename all arguments + """ + + CHOICES = '0', '1', '2', '3' + + +class RftNamePattern(TextOption): + """Match ast.Name in ast.Tree + + Each pattern includes 2 parts: + + - ModulePattern: fnmatchcase pattern, match module + + - NamePattern: fnmatchcase pattern, match function/class/name + """ + pass + + +class RftAttrFilter(ListOption): + """Filter ast.Attribute transformed to setattr/getattr + + Each ruler include 3 patterns + + - ModulePattern: fnmatchcase pattern, match module + + - ScopePattern: fnmatchcase pattern, match function/class + + - AttributePattern: fnmatchcase pattern, match attribute + + a: match any attribute "a" + b.a: match only "b.a" + a*: match any attribute startswith "a" + b.a*: match any "b" attribute which startswith "a" + b*.a: match any attribute "a" in parent startswith "b" + ().a: parent is function call + [].a: parent is subscript + + *.a: same as "a" + """ + pass + + +SECTIONS = { + 'pyarmor': [], + 'finder': [], + 'builder': [], + 'pack': [], + + # Group: filter + 'assert.import': [], + 'assert.call': [], + 'mix.str': [], + + # Group bcc + 'bcc': [], + + # Scope project + 'project': [ + + OptionModel( + name='src', + cls='PathOption', + examples=[ + '', '.', 'src', '../src', + '/home/jondy/project/src1', + 'my projects/project/src4' + 'C:\\test\\src2', 'C:\\工程\\src3', + ] + ), + + OptionModel( + name='scripts', + cls='FileOption', + many=True, + restricts=[ + 'ext should be .py, .pyw', + 'relative to src, or absolute path', + ], + examples=['foo.py', 'test/run.py'] + ), + + OptionModel( + name='modules', + cls='FileOption', + many=True, + ), + + OptionModel( + name='packages', + cls='PkgPathOption', + many=True, + ), + + # Only match name, no path separator + OptionModel( + name='excludes', + cls='PatternOption', + many=True, + examples=['__pycache__', 'test*'] + ), + + OptionModel( + name='recursive', + cls='BoolOption', + value=0, + ), + OptionModel( + name='pypaths', + cls='PatternOption', + many=True, + ), + ], + + 'rft': [ + OptionModel( + name='remove_assert', + cls='BoolOption', + value=0, + ), + OptionModel( + name='remove_docstr', + cls='BoolOption', + value=0, + ), + OptionModel( + name='builtin_mode', + cls='BoolOption', + value=0, + ), + OptionModel( + name='argument_mode', + cls='RftArgEnum', + value='3', + ), + OptionModel( + name='export_mode', + cls='BoolOption', + value=0, + ), + OptionModel( + name='extra_builtins', + cls='NameOption', + many=True, + ), + OptionModel( + name='exclude_names', + cls='RftNamePattern', + many=True, + ), + OptionModel( + name='exclude_funcs', + cls='RftNamePattern', + many=True, + ), + OptionModel( + name='external_types', + cls='RftNamePattern', + many=True, + ), + OptionModel( + name='external_attrs', + cls='RftNamePattern', + many=True, + ), + OptionModel( + name='attr_rules', + cls='RftAttrFilter', + ), + OptionModel( + name='call_rules', + cls='RftAttrFilter', + ), + OptionModel( + name='var_types', + cls='DictOption', + ), + ], + 'rft_filter': [ + OptionModel( + name='obf_attribute', + cls='BoolOption', + ), + OptionModel( + name='obf_string', + cls='BoolOption', + ), + OptionModel( + name='obf_include_strings', + cls='PatternOption', + many=True, + ), + OptionModel( + name='obf_attr_filters', + cls='RftAttrFilter', + ), + OptionModel( + name='rft_import', + cls='BoolOption', + ), + OptionModel( + name='rft_ximport', + cls='BoolOption', + ), + OptionModel( + name='wildcard_import_table', + cls='DictOption', + ), + ], + 'mini': [ + OptionModel( + name='import_from', + cls='NameOption', + value='pyarmor.mini.pyarmor_mini', + ), + ], + 'vmc': [], + 'ecc': [], +} + + +GROUPS = { + 'filter': ('assert.import', 'assert.call', 'mix.str'), + 'bcc': ('bcc', + 'linux.x86_64.bcc', 'linux.x86.bcc', + 'linux.aarch64.bcc', 'linux.armv7.bcc', + 'darwin.x86_64.bcc', 'darwin.aarch64.bcc', + 'windows.x86_64.bcc', 'windows.x86.bcc', + 'android.x86_64.bcc', 'android.x86.bcc', + 'android.aarch64.bcc', 'android.armv7.bcc', + 'alpine.x86_64.bcc', 'alpine.aarch64.bcc', + 'freebsd.x86_64.bcc'), +} + +BUSINESS_MODELS = { + + 'Project': [ + Item('scripts', cls='Python.Script', many=True), + Item('modules', cls='Python.Module', many=True), + Item('packages', cls='Python.Package', many=True), + ], + + 'StdScript': [ + Item('script', cls='Python.Script', require=True), + Item('pyarmor_runtime', cls='Python.Extension', require=True), + ], + + 'MiniScript': [ + Item('script', cls='Python.Script', require=True), + Item('pyarmor_mini', cls='Python.Extension', require=True), + ], + + 'EccScript': [ + Item('script', cls='Python.Script', require=True), + Item('pyarmor_ecc', cls='Python.Extension', require=True), + ], + + 'VmcScript': [ + Item('script', cls='Python.Script', require=True), + Item('pyarmor_ecc', cls='Python.Extension', require=True), + ], + + 'RftScript': [ + Item('script', cls='Python.Script', require=True), + ] +} diff --git a/src/cli/plugin.py b/src/cli/plugin.py index 7bb76ee1..b6a696c3 100644 --- a/src/cli/plugin.py +++ b/src/cli/plugin.py @@ -186,7 +186,7 @@ def post_build(ctx, inputs, outputs, pack): os.makedirs(verpath) pkgscript = os.path.join(pkgpath, '__init__.py') - with open(pkgscript) as f: + with open(pkgscript, encoding=ctx.encoding) as f: lines = f.readlines() start = 1 if lines[0].startswith('#') else 0 @@ -196,13 +196,13 @@ def post_build(ctx, inputs, outputs, pack): '{0} = __import__("py%d%d.pyarmor_runtime" % py_version[:2],' ' globals(), locals(), ["{0}"], 1).{0}'.format('__pyarmor__') ]) - with open(pkgscript, 'w') as f: + with open(pkgscript, 'w', encoding=ctx.encoding) as f: f.write(''.join(lines)) for x in MultiPythonPlugin.RUNTIME_FILES: move(x, verpath) else: lines[start:start] = 'from sys import version_info as py_version\n' - with open(pkgscript, 'w') as f: + with open(pkgscript, 'w', encoding=ctx.encoding) as f: f.write(''.join(lines).replace( "join(['_'", "join(['py%d%d' % py_version[:2], '_'")) for x in MultiPythonPlugin.RUNTIME_FILES: @@ -249,8 +249,11 @@ class DarwinUniversalPlugin: def post_build(ctx, inputs, outputs, pack): from shutil import rmtree + if not ctx.native_platform.startswith('darwin.'): + return + def rebuild_init(oneplat, init_script): - with open(init_script, 'r') as f: + with open(init_script, 'r', encoding=ctx.encoding) as f: lines = f.readlines() if oneplat: lines[1:] = ['from .pyarmor_runtime import __pyarmor__'] @@ -259,7 +262,7 @@ def rebuild_init(oneplat, init_script): if lines[i].strip().startswith("# mach = 'universal'"): lines[i] = lines[i].replace('# ', '') break - with open(init_script, 'w') as f: + with open(init_script, 'w', encoding=ctx.encoding) as f: f.write(''.join(lines)) rtpath = find_runtime_package(ctx, outputs[0]) diff --git a/src/cli/project.py b/src/cli/project.py new file mode 100755 index 00000000..fd4b0a62 --- /dev/null +++ b/src/cli/project.py @@ -0,0 +1,1281 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- +# +############################################################# +# # +# Copyright @ 2024 Dashingsoft corp. # +# All rights reserved. # +# # +# pyarmor # +# # +# Version: 9.1.0 - # +# # +############################################################# +# +# +# @File: pyarmor/cli/project.py +# +# @Author: Jondy Zhao(pyarmor@163.com) +# +# @Create Date: Tue Nov 12 16:38:51 CST 2024 +# +# @Description: +# +# - Define project object for Pyarmor 9. +# - Define project commands: init, build +# - Define targets: std, mini, vmc, ecc, rft + +"""Manage projects + +Config View +----------- + +[project] +global_excludes = .* __pycache__ +project_includes = *.py *.pyw +package_includes = *.py *.pyw +recursive = 0 + +name = str +src = absolute path + +modules = pattern pattern ... +packages = pattern path@name @section ... +excludes = patterh patterh ... + +pypaths = path modname::path + +Examples +-------- + +1. Create a project with scripts/packages in current path + + $ pyarmor init -r + +2. Obfuscate all the scripts in the project + + $ pyarmor build --rft + $ pyarmor build --mini + +3. Config project and print project information + + $ pyarmor env -p + $ pyarmor env -p set rft:remove_docstr 1 + +""" +import ast +import glob +import logging +import os +import tokenize + +from collections import namedtuple +from fnmatch import fnmatch +from json import loads as jsonloads, load as jsonload +from os.path import ( + abspath, basename, exists, isabs, join as joinpath, + normpath, relpath, splitext +) +from string import Template +from textwrap import dedent + + +logger = logging.getLogger('cli.build') + + +GRAPHVIZ_INDENT = ' ' + +############################################################ +# +# Project File View +# +############################################################ + +GLOBAL_EXCLS = '.*', '__pycache__' +GLOBAL_INCLS = '*.py', '*.pyw' + +ProjectItem = namedtuple( + 'ProjectItem', + ('name', 'src', 'scripts', 'modules', 'packages', + 'excludes', 'recursive'), + defaults=['', '', [], [], [], None, False] +) + + +def scan_path(path, includes=None, excludes=[], **options): + files, dirs = [], [] + xlist = includes if includes else GLOBAL_INCLS + with os.scandir(path) as itdir: + for et in itdir: + if any([fnmatch(et.name, x) for x in excludes]): + continue + if et.is_dir(follow_symlinks=False): + dirs.append(et.name) + elif (et.is_file(follow_symlinks=False) and + any([fnmatch(et.name, x) for x in xlist])): + files.append(et.name) + return files, dirs + + +def search_item(root, pattern, excludes, recursive=0): + if not pattern: + return [] + + sep = os.sep if pattern.endswith(os.sep) else '' + result = [] + + pt = pattern if isabs(pattern) else joinpath(root, pattern) + for item in glob.glob(pt, recursive=recursive): + name = basename(item.strip(sep)) + if excludes and any([fnmatch(name, x) for x in excludes]): + continue + result.append(item) + return [normpath(x) for x in result] + + +############################################################ +# +# Concepts +# +############################################################ + +class Module: + """Module concept""" + + def __init__(self, path, name=None, parent=None): + self.parent = parent + self._path = path + self._name = name if name else splitext(basename(path))[0] + + self._co = None + self._tree = None + self._type = None + + self._shebang = '' + + @property + def name(self): + return '' if self._name == '__init__' else self._name + + @property + def path(self): + return self._path + + @property + def mtype(self): + return self._type + + @property + def mtree(self): + return self._tree + + @property + def qualname(self): + if isinstance(self.parent, (Project, type(None))): + return self._name + prefix = self.parent.qualname + ('.' if self.name else '') + return prefix + self.name + + @property + def project(self): + """Return project this module belong to""" + return (self.parent if isinstance(self.parent, Project) + else self.parent.project) + + @property + def abspath(self): + return (self._path if isabs(self._path) else + joinpath(self.parent.abspath, self._path)) + + @property + def destpath(self): + s = self.qualname + ('' if self.name else '.__init__') + return joinpath(*s.split('.')) + splitext(self.path)[-1] + + @property + def shebang(self): + return self._shebang + + def compile_file(self, force=False, optimize=-1): + if self._co is not None and not force: + return + + self.parse_file(force=force) + + options = { + 'optimize': optimize + } + logger.info('compile %s ...', self.qualname) + self._co = compile(self._tree, self.abspath, 'exec', **options) + logger.info('compile %s end', self.qualname) + + def parse_file(self, force=False): + if self._tree is not None and not force: + return + + filename = self.abspath + with open(filename, 'rb') as f: + encoding, lines = tokenize.detect_encoding(f.readline) + if lines and lines[0].startswith(b'#!'): + self._shebang = lines[0].decode(encoding) + + with open(filename, 'r', encoding=encoding) as f: + logger.info('parse %s ...', self.qualname) + self._tree = ast.parse(f.read(), filename, 'exec') + logger.info('parse %s end', self.qualname) + + def _as_dot(self): + return self.name + + +class Script(Module): + """Script concept""" + + @property + def rpaths(self): + """Extra Python paths for importing module + + If one script has any extra pypath, it need map imported + module name to project module name + + For example, in the script `import abc`, maybe it uses module + `pkg.abc` in this project + + This is only used by RFT mode, otherwise it doesn't know where + to find module `abc` + + This property is used to generate internal `_mapped_modules` + """ + pass + + +class Package(Module): + """Package concept""" + + def __init__(self, path, name=None, parent=None, excludes=[]): + super().__init__(path, name=name, parent=parent) + + self._modules = None + self._packages = None + self._excludes = excludes if excludes else GLOBAL_EXCLS + self._filters = None + + @property + def filters(self): + if self._filters is None: + self._filters = [] + for x in self._excludes: + i = x.find(':') + if i == -1: + self._filters.append(x) + elif fnmatch(self.qualname, x[:i]): + self._filters.append(x[i+1:]) + return self._filters + + def load(self): + excls = self.filters + files, dirs = scan_path(self.abspath, excludes=excls) + self._modules = [Module(x, parent=self) for x in files] + self._packages = [ + Package(x, parent=self, excludes=self._excludes) + for x in dirs + ] + + @property + def modules(self): + """Each package has many modules + + There is one special module `__init__` for Package + """ + if self._modules is None: + self.load() + + for x in self._modules: + yield x + + @property + def packages(self): + """Each package has many sub-packages""" + if self._packages is None: + self.load() + + for x in self._packages: + yield x + + def iter_module(self): + if self._modules is None or self._packages is None: + self.load() + + for x in self._modules: + yield x + + for pkg in self._packages: + for x in pkg.iter_module(): + yield x + + def _as_dot(self, n=0): + modules = [x._as_dot() for x in self.modules] + packages = [x._as_dot(n+1) for x in self.packages] + sep = '\n' + GRAPHVIZ_INDENT + source = Template(dedent("""\ + subgraph cluster_$cid { + label="$name"; + $modules + $packages + }""")).substitute( + cid=id(self), + name=self.name, + modules=sep.join(modules), + packages=sep.join(packages), + ) + return ( + ('\n' + GRAPHVIZ_INDENT * n).join(source.splitlines()) + if n else source + ) + + +class Namespace: + """Namespace concept""" + + @property + def name(self): + pass + + @property + def components(self): + """Each component has 3 items: path, modules, children + + Each child is Namespace or Package + """ + return [] + + +class Project: + """Project conpect + + Project is compose of Python elements and obfuscation settings + + Each project has 4 components: + + - Script + - Module + - Package + - Namespace + + Each component has one unique name in this project except Script + + It may has alias which also can't be duplicated with other names + + Refer + ----- + + https://docs.python.org/3.13/reference/import.html#namespace-packages + + """ + + ATTR_LOGFILE = '.pyarmor/project/rft_unknown_attrs.log' + CALL_LOGFILE = '.pyarmor/project/rft_unknown_calls.log' + + def __init__(self, ctx): + self.ctx = ctx + self.src = '' + + self._scripts = [] + self._modules = [] + self._packages = [] + self._namespaces = [] + + self._rft_options = None + self._rft_filters = None + self._rft_rulers = None + + self._rmodules = None + self._builtins = None + + self._rft_type_rules = None + self._rft_include_attrs = None + self._used_external_types = None + + self._std_options = None + self._mini_options = None + self._vmc_options = None + self._ecc_options = None + + # Log variable name in chain attributes + # + # For example, in module "foo.py": + # + # def fa(x): + # x.runner[0].start() + # + # Because don't know the type of "x", log it as + # + # self.unknown_vars.append("foo:fa:x") + # + # self.unknown_vars = [] + + # Log attribute used but not defined in class + # + # For example, in module "foo.py": + # + # def fa(x: Fibo): + # x.items[0].run() + # + # If no found attribute "items" in class "Fibo", log it as + # + # self.unknown_attrs.append("foo:fa:x.items.run ?items") + # + self.unknown_attrs = set() + + # Log function which called with **kwargs + # + # For example, in the module "foo.py": + # + # class Fibo: + # + # def runner(self, a=1, b=2): + # return a + b + # + # c = Fibo() + # c.runner(**data) + # + # Because the method "Fibo.runner" is called by dict + # argument "**data", log it as + # + # self.unknown_funcs.append("foo:Fibo.runner") + # + self.unknown_funcs = [] + + # Log unknown caller with keyword arguments. + # + # For example, in the module "foo.py": + # + # def fa(c): + # c.runner[2].echo(msg='hello') + # + # If don't know where "echo" is defined, log it as + # + # self.unknown_calls.append("foo:fa:c.runner.echo") + # + # If it uses dict arguments, for example: + # + # def fa(c): + # c.runner[2].echo(**data) + # + # Log it with suffix "*" + # + # self.unknown_calls.append("foo:fa:c.runner.echo*") + # + self.unknown_calls = None + + # Log all attributes which is external base class attr + # + # For example: + # + # class C(dict): + # + # def merge(self, another): + # super().merge(another) + # + # "merge" will be loged to external_attrs + # self.external_attrs = [] + + @property + def abspath(self): + return abspath(self.src) + + @property + def scripts(self): + """Project entry points + + One project may has many entry points + + Script can't be imported by other components + """ + for x in self._scripts: + yield x + + @property + def modules(self): + for x in self._modules: + yield x + + @property + def packages(self): + """Only top packages""" + for x in self._packages: + yield x + + @property + def namespaces(self): + """Only top namespace""" + for x in self._namespaces: + yield x + + @property + def std_options(self): + """Options got from ctx.builder, not in project""" + if self._std_options is None: + cfg = self.ctx.cfg + optimize = cfg['builder'].getint('optimize', -1) + self._std_options = { + 'optimize': optimize + } + return self._std_options + + def std_opt(self, name): + return self.std_options.get(name) + + @property + def rft_options(self): + """Refactor options: + + - remove_assert: bool + + If 1, remove assert node + + - remove_docstr: bool + + If 1, remove all docstring + + - builtin_mode: bool + + 0, do not touch any builtin names + 1, as build target, maybe std, mini or plain + + - rft_import: bool + + always 1 + + - rft_ximport: bool + + Always 0, do not touch node "from..import *" + + - argument_mode: enum('0', '1', '2', '3') + + 0: "no", no reform any argument node + 1: "pos", reform posonly arguments + 2: '!kw', no reform keyword only arguments + 3: "all", reform all arguments + + Note that if function is exported, no arguments reformed + + - obf_attribute: enum(no, yes, all) + + Reform attribute node to setattr() or getattr() + + 0, do not reform attribute to setattr or getattr + 1, reform attribute node by rft_attribute_filters + 2, reform all attribute node + + - obf_string: enum(no, yes, all) + + Reform string constant to security mode + + It only works for std/mini target + + It's always 0 for plain target + + 0, no reform string + 1, reform string by rft_string_filters + 2, reform all string + + - export_mode: bool + + True: auto export all names in module.__all__ + + If class is exported, all class members are exported + If function is exported, arguments can't be renamed + Only module variable can be exported separately + + - exclude_names: list + + Move it from rft_filter + + - exclude_funcs: list + + Move it from rft_filter + + - rft_str_keywords: list (not implemented) + + Rename string constant or key in dict constant + + When call function, solve argument not found issue + + - rft_type_rules: dict + + Specify variable type + + - extra_type_info: dict + + Specify extra attribute for module or type + + - wildcard_import_table: dict (unused now) + + When building target, it need import module to get name + for wildcard import, sometimes it may failed + + In order to avoid importing module in build time, users + can provide all names in wildcard imported module + + - extra_builtins: list + + By default, builtin names is got from builtin module + + User can append extra builtin names + + - on_unknown_attr: enum(ask, log, yes, no, err) + + When don't know how to refactor attribute + + ask: query user interactively + log: no touch attr but log it + yes: rename attr + no: do thing + err: raise error + """ + if self._rft_options is None: + cfg = self.ctx.cfg + sect = 'rft' + if cfg.has_section(sect): + self._rft_options = dict(cfg.items(sect)) + else: + self._rft_options = {} + return self._rft_options + + def rft_opt(self, name): + return self.rft_options.get(name) + + @property + def rft_exclude_names(self): + """Exclude module, class, function + + All names in this scope aren't renamed. For example, + + Exclude module, all classes and functions aren't renamed + Exclude class, all class attributes aren't renamed + + Each ruler is one chained names + Each ruler must start with package or module name + It supports pattern match as fnmatchcase + Pattern only match one level + """ + value = self.rft_opt('exclude_names') + if value: + for x in value.splitlines(): + yield x + + @property + def rft_exclude_funcs(self): + """No touch arguments for listed functions""" + value = self.rft_opt('exclude_funcs') + if value: + for x in value.splitlines(): + yield x + + @property + def rft_filters(self): + if self._rft_filters is None: + cfg = self.ctx.cfg + sect = 'rft_filter' + if cfg.has_section(sect): + self._rft_filters = dict(cfg.items(sect)) + else: + self._rft_filters = {} + return self._rft_filters + + @property + def obf_include_strings(self): + """A list of re pattern based on obf_string + + All matched string in ast.Tree will be transformed + """ + value = self.rft_filters.get('obf_include_strings', '') + for x in value.splitlines(): + yield x + + @property + def obf_attr_filters(self): + """A list of re pattern based on obf_attribute + + All matched ast.Attribute will be transformed to call + setattr() or getattr() to hide attribute name + """ + value = self.rft_filters.get('obf_attr_filters', '') + for x in value.splitlines(): + yield x + + @property + def rft_rulers(self): + if self._rft_rulers is None: + cfg = self.ctx.cfg + sect = 'rft_ruler' + if cfg.has_section(sect): + self._rft_rulers = dict(cfg.items(sect)) + else: + self._rft_rulers = {} + return self._rft_rulers + + @property + def rft_attr_rules(self): + """Refactor attribute rules, for special attribute node + + If can't decide variable type, use rule for chains + + For example, "x.a.b", if "x" of type is unknown + + Use rule "x.a.b" to rename attribute "a", "b" + + Use rule "x.a.b.c" to rename "a", "b", "c" + + Use rule "x.a.b -> *.?.*" to rename "a" only + + Use rule "*.write -> *.write" to keep all write attribute + """ + value = self.rft_options.get('attr_rules', '') + for x in value.splitlines(): + yield x + + @property + def rft_call_rules(self): + """Refactor keyword argument in call statement + + If can't decide function type, use ruler to rename arg + """ + value = self.rft_options.get('call_rules', '') + for x in value.splitlines(): + yield x + + @property + def rft_arg_rules(self): + """Refactor rule, for arg name in Function/Call node + + For example, in the call statement + + kwargs = { 'msg': 'hello' } + foo(**kwargs) + + This kind of rule could be used to rename string `msg` + """ + value = self.rft_options.get('rft_arg_rules', '') + for x in value.splitlines(): + yield x + + @property + def rft_type_rules(self): + """Specify variable/blockvar type + + Support format: + + modname:scope:var typename + modname:scope:var.[?] typename + modname:scope:func.() typename + modname:scope:cls.method.() typename + + For blockvar in For/Comprehension/With + modname:scope:{var} typename + """ + if self._rft_type_rules is None: + vartypes = {} + lines = self.rft_opt('var_types') + for line in lines.splitlines() if lines else []: + varinfo, tname = line.split() + if varinfo.endswith('}'): + info = varinfo[:-1].replace('{', '') + modname, varname = info.split(':', 1) + vartypes.setdefault(modname, {}) + mtypes = vartypes[modname] + mtypes.setdefault('__ivars__', {}) + mtypes['__ivars__'][varname] = tname + else: + modname, varname = varinfo.split(':', 1) + vartypes.setdefault(modname, {}) + vartypes[modname][varname] = tname + self._rft_type_rules = vartypes + return self._rft_type_rules + + @property + def mini_options(self): + """MINI mode options: + + - import_from: str + + Default value is "pyarmor.mini.pyarmor_mini", it means from + package `pyarmor.mini` import extension `pyarmor_mini` + + Set it to "pyarmor_mini" if it's distributed with obfuscated + scripts + """ + if self._mini_options is None: + cfg = self.ctx.cfg + sect = 'mini' + if cfg.has_section(sect): + self._mini_options = dict(cfg.items(sect)) + else: + self._mini_options = {} + self._mini_options.setdefault( + 'import_from', 'pyarmor.mini.pyarmor_mini') + return self._mini_options + + def mini_opt(self, name): + return self.mini_options.get(name) + + @property + def vmc_options(self): + """VMC mode options""" + if self._vmc_options is None: + cfg = self.ctx.cfg + sect = 'vmc' + if cfg.has_section(sect): + self._vmc_options = dict(cfg.items(sect)) + else: + self._vmc_options = {} + return self._vmc_options + + def vmc_opt(self, name): + return self.vmc_options.get(name) + + @property + def ecc_options(self): + """ECC mode options""" + if self._ecc_options is None: + cfg = self.ctx.cfg + sect = 'ecc' + if cfg.has_section(sect): + self._ecc_options = dict(cfg.items(sect)) + else: + self._ecc_options = {} + return self._ecc_options + + def ecc_opt(self, name): + return self.ecc_options.get(name) + + @property + def builtins(self): + if self._builtins is None: + import builtins + self._builtins = set(dir(builtins)) + return self._builtins + + @property + def used_external_types(self): + """Used external types""" + if self._used_external_types is None: + used_types = {} + names = 'builtins' + for name in names.split(): + if name.endswith('.json'): + if exists(name): + with open(name, encoding='utf-8') as f: + used_types.update(jsonload(f)) + else: + modname = name.split('::')[0] + alltypes = self._get_external_types(modname) + if alltypes: + used_types.update(alltypes) + self._used_external_types = used_types + return self._used_external_types + + @property + def rft_external_types(self): + """External types manual""" + value = self.rft_options.get('external_types', '') + if value: + for x in value.split(): + yield x + + @property + def rft_external_attrs(self): + """External attrs manual + x means append to attrs of rft_external_types + !x means exclude x from attrs of rft_external_types + """ + value = self.rft_options.get('external_attrs', '') + if value: + for x in value.split(): + yield x + + @property + def rft_include_attrs(self): + """List attributes need to be renamed on unknown type""" + if self._rft_include_attrs is None: + self._rft_include_attrs = set() + value = self.rft_options.get('include_attrs', '') + if value: + self._rft_include_attrs.update(value.split()) + return self._rft_include_attrs + + def get_module(self, qualname): + """Get module in the project by unique qualname + It equals one dict: map_qualname_to_module + """ + if self._rmodules is None: + self._rmodules = { + x.qualname: x for x in self.iter_module() + } + return self._rmodules.get(qualname) + + def iter_module(self): + """Iterate all modules in this project""" + for x in self._scripts: + yield x + + for x in self._modules: + yield x + + for child in self._packages + self._namespaces: + for x in child.iter_module(): + yield x + + def relsrc(self, path): + return relpath(path, self.src) + + def load(self, data): + """Init project object with dict + + It equals: + + 1. map init data to ProjectItem + 2. map ProjectItem to project files + """ + dp = joinpath(self.ctx.local_path, 'project') + os.makedirs(dp, exist_ok=True) + + def vlist(name): + return [x.strip().replace('%20%', ' ') + for x in data.get(name, '').split()] + + src = self.src = data['src'] + name = data.get('name') + excludes = vlist('excludes') + list(GLOBAL_EXCLS) + proexcls = [x.strip(':') for x in excludes + if x.find(':') < 1] + + scripts = [] + for pat in vlist('scripts'): + scripts.extend(search_item(src, pat, proexcls)) + self._scripts.extend([ + Script(self.relsrc(x), parent=self) for x in scripts + ]) + + modules = [] + for pat in vlist('modules'): + modules.extend(search_item(src, pat, proexcls)) + + packages = vlist('packages') + if packages: + for item in packages: + # 3 forms: path, path@name, @sect + i = item.find('@') + if i == 0: + raise NotImplementedError(f'package {item}') + if i == -1: + path, pkgname = item, basename(item) + else: + path, pkgname = item.split('@') + if not isabs(path): + path = joinpath(src, path) + obj = Package(path, + name=pkgname, + parent=self, + excludes=excludes) + self._packages.append(obj) + + recursive = data.get('recursive', '0') + if recursive == '1': + pkginit = joinpath(src, '__init__.py') + if exists(pkginit): + pkgname = name if name else basename(src) + obj = Package(src, + name=pkgname, + parent=self, + excludes=excludes) + self._packages.append(obj) + else: + files, dirs = scan_path(src, excludes=proexcls) + modules.extend([joinpath(src, x) for x in files]) + self._packages.extend([ + Package(x, parent=self) for x in dirs + ]) + + if scripts and modules: + for x in set(scripts) & set(modules): + logger.debug('duplicated %s', self.relsrc(x)) + modules.remove(x) + self._modules.extend([ + Module(self.relsrc(x), parent=self) for x in modules + ]) + + logger.info('load %d scripts', len(self._scripts)) + logger.info('load %d modules', len(self._modules)) + logger.info('load %d packages', len(self._packages)) + + def start(self): + self._logfile = open(self.ATTR_LOGFILE, 'w') + self._logfile2 = open(self.CALL_LOGFILE, 'w') + + def stop(self): + self._logfile.close() + self._logfile2.close() + + def log_unknown_attr(self, line): + fields = line.split(':') + attrs = fields[2].split('.') + start = int(fields[3]) + self.unknown_attrs.update([ + x for x in attrs[start:] if x[:1] not in ('(', '[') + ]) + self._logfile.write(line + '\n') + + def log_unknown_func(self, func): + if func not in self.unknown_funcs: + self.unknown_funcs.append(func) + + def log_unknown_call(self, line): + self.unknown_calls = True + self._logfile2.write(line + '\n') + + def _get_external_types(self, modname, pypaths=None): + from sys import executable + from subprocess import check_output, CalledProcessError, DEVNULL + source = Template(dedent("""\ + import json + import sys + x = ${pypaths} + if x: + sys.path[0:0] = [x] if isinstance(x, str) else x + import ${name} + typeinfo = {'${name}': []} + for key, value in ${name}.__dict__.items(): + if key[:2] == '__': + continue + if isinstance(value, type): + typeinfo['${name}.%s' % key] = [ + x for x in dir(value) if x[:2] != '__' + ] + typeinfo['${name}'].append(key) + print(json.dumps(typeinfo)) + """)).substitute(name=modname, pypaths=repr(pypaths)) + + try: + output = check_output([executable, '-c', source], stderr=DEVNULL) + return jsonloads(output) + except CalledProcessError: + pass + + def get_external_type(self, modname, clsname=None): + if not all([x.isidentifier() for x in modname.split('.')]): + logger.debug('invalid external module: %s', modname) + return [] + + extypes = self.used_external_types + if modname not in extypes: + result = self._get_external_types(modname) + if result: + extypes.update(result) + if clsname is None: + return extypes + return extypes.get('%s.%s' % (modname, clsname), []) + + def preview_autofix_result(self, mode): + if mode not in (2, 3): + return + output = f'.pyarmor/project/rft_autofix.{mode}.org' + + rulefile = '.pyarmor/project/rft_autofix.rules' + if not exists(rulefile): + logger.info('no found %s', rulefile) + return + with open(rulefile, encoding='utf-8') as f: + fixtable = jsonload(f) + + if mode == 2: + self._preview_autofix_2(fixtable, output) + elif mode == 3: + self._preview_autofix_3(fixtable, output) + + def _preview_autofix_2(self, fixtable, output): + modules = fixtable['modules'] + attrinfo = {} + with open(self.ATTR_LOGFILE) as f: + # Line format: + # modname:scopes:attrs:index:begin_line,end_line + for line in f: + line = line.strip() + fields = line.split(':') + modname, scope = fields[:2] + attrs = [x for x in fields[2].split('.') + if x[:1] not in ('(', '[')] + start = int(fields[3]) + lineno = [int(x) for x in fields[4].split(',')] + for s in attrs[start:]: + attrinfo.setdefault(s, []) + fname = modules.get(modname, modname) + title = ':'.join([ + modname, scope, '.'.join(attrs) + ]) + attrinfo[s].append([fname, lineno[0], title]) + + header = Template(dedent("""\ + * RFT AutoFix Mode 2 + + The following attributes have been renamed, if any of + them should not be renamed, configure it as external + attribute and run autofix mode again. + + For example, suppose `append` is external attribute: + + pyarmor env -p push rft:external_attrs append + pyarmor build --autofix 2 + pyarmor build --rft + + Refactoring Attribute List + + $attrs + + """)) + attrsect = Template(dedent("""\ + ** $name + + $srclinks + + """)) + + inattrs = fixtable.get('include_attrs', []) + rftattrs = header.substitute(attrs='\n'.join( + [f'[[*{x}][{x}]]' for x in inattrs])) + with open(output, 'w', encoding='utf-8') as fp: + fp.write(rftattrs) + for attr in inattrs: + infos = attrinfo.get(attr) + if infos is None: + continue + fp.write(attrsect.substitute( + name=attr, + srclinks='\n'.join([ + f'- [[file:{fname}::{n}][{title}]]' + for fname, n, title in infos + ]) + )) + + def _preview_autofix_3(self, fixtable, output): + modules = fixtable['modules'] + confused_names = fixtable.get('confused_names', []) + attrules = {} + rftattrs = {} + with open(self.ATTR_LOGFILE) as f: + # Line format: + # modname:scopes:attrs:index:begin_line,end_line + for line in f: + fields = line.strip().split(':') + modname, scope = fields[:2] + attrs = [x for x in fields[2].split('.') + if x[:1] not in ('(', '[')] + start = int(fields[3]) + lineno = fields[4].split(',')[0] + actions = ['*'] * start + rnames = [] + for s in attrs[start:]: + if s in confused_names: + actions.append('?') + rnames.append(s) + else: + actions.append('*') + if rnames: + aname = '.'.join(attrs) + pat = '.'.join(actions) + filename = modules.get(modname, modname) + rule = f'{modname}::{scope}:{aname} {pat}' + anchor = modname, filename, lineno + if rule not in attrules: + attrules[rule] = [anchor] + else: + attrules[rule].append(anchor) + s = rnames[0] + rftattrs.setdefault(s, set()) + rftattrs[s].add(rule) + + header = Template(dedent("""\ + * RFT AutoFix Mode 3 + + The following attributes have been renamed, if any of + them should not be renamed, add one attribute rule and + run autofix mode again. + + For example, there is one line + + *** rftbuild::log_patch_script:cfgpatch.append *.? + - rftbuild.py::2690 + + But `cfgpatch` is builtin type `list`, and its attribute + `append` shouldn't be renamed + + In this case, replace `?` with `*`, add one rule: + + pyarmor env -p push rft:attr_rules "rftbuild::log_patch_script:cfgpatch.append *.*" + + Then rebuild the project: + + pyarmor build --autofix 3 + pyarmor build --rft + + Refactoring Attribute List + + $attrs + + """)) + attrsect = Template(dedent("""\ + ** $name + + $rules + + """)) + + rulesect = Template(dedent("""\ + *** $rule + $srclinks + """)) + + attrs = sorted(rftattrs.keys()) + with open(output, 'w', encoding='utf-8') as fp: + fp.write(header.substitute( + attrs='\n'.join([f'[[*{x}][{x}]]' for x in attrs]) + )) + for key in attrs: + rules = rftattrs[key] + rlist = [] + for r in rules: + infos = attrules[r] + rlist.append(rulesect.substitute( + rule=r, + srclinks='\n'.join([ + f'- [[file:{s}::{n}][{m}:{n}]]' + for m, s, n in infos + ]) + )) + fp.write(attrsect.substitute( + name=key, + rules='\n'.join(rlist) + )) + + def _as_dot(self): + """Map project to dot graph""" + modules = [x._as_dot() for x in self.modules] + packages = [x._as_dot() for x in self.packages] + sep = '\n' + GRAPHVIZ_INDENT * 2 + return Template(dedent("""\ + graph { + layout=osage + subgraph cluster_0 { + label="Project Structure"; + $modules + $packages + } + }""")).substitute( + modules=sep.join(modules), + packages=sep.join('\n'.join(packages).splitlines()) + ) + + +if __name__ == '__main__': + pass diff --git a/src/cli/register.py b/src/cli/register.py index c5083957..628dd49e 100644 --- a/src/cli/register.py +++ b/src/cli/register.py @@ -21,7 +21,7 @@ # import os -from base64 import b64decode, urlsafe_b64encode +from base64 import b64decode, b64encode, urlsafe_b64encode from json import loads as json_loads from string import Template @@ -31,48 +31,107 @@ # All supported machine flags for group license: [11, 26) MACHFLAGS = 22, 21, 18, 20, 16, 11 +# Upgrade notes for Pyarmor 9 +URL_UPGRADE_V9 = 'https://github.com/dashingsoft/pyarmor/issues/1958' + +# Template for license info +LICENSE_INFO_TEMPLATE = '''$advanced + +$notes +''' + def parse_token(data): from struct import unpack - if not data or data.find(b' ') == -1: - return { - 'token': 0, - 'rev': 0, - 'features': 0, - 'licno': 'pyarmor-vax-000000', - 'regname': '', - 'product': 'non-profits', - 'note': 'This is trial license' - } - - buf = b64decode(data.split()[0]) - - token, value = unpack('II', buf[:8]) - rev, features = value & 0xff, value >> 8 - licno = buf[16:34].decode('utf-8') - - pstr = [] - i = 64 - for k in range(4): - n = buf[i] - i += 1 - pstr.append(buf[i:i+n].decode('utf-8') if n else '') - i += n - - product = 'non-profits(TBD)' if pstr[2] in ('', 'TBD') else pstr[2] + if data and data.find(b' ') > 0: + try: + buf = b64decode(data.split()[0]) + + token, value = unpack('II', buf[:8]) + rev, features = value & 0xff, value >> 8 + licno = buf[16:34].decode('utf-8') + + pstr = [] + i = 64 + for k in range(4): + n = buf[i] + i += 1 + pstr.append(buf[i:i+n].decode('utf-8') if n + else '') + i += n + + product = ('non-profits(TBD)' if pstr[2] in ('', 'TBD') + else pstr[2]) + return { + 'token': token, + 'rev': rev, + 'features': features, + 'licno': licno, + 'machine': pstr[0], + 'regname': pstr[1], + 'product': product, + 'note': pstr[3], + } + except Exception as e: + logger.warning('bad token: %s', str(e)) + return { - 'token': token, - 'rev': rev, - 'features': features, - 'licno': licno, - 'machine': pstr[0], - 'regname': pstr[1], - 'product': product, - 'note': pstr[3], + 'token': 0, + 'rev': 0, + 'features': 0, + 'licno': 'pyarmor-vax-000000', + 'regname': '', + 'product': 'non-profits', + 'note': 'This is trial license' } +def show_help_page(prompt, url): + choice = input('\n'.join(prompt)).lower()[:1] + if choice in ('h', 'y'): + import webbrowser + webbrowser.open(url) + return choice + + +def check_license_version(ctx, silent=False): + licinfo = ctx.license_info + rev = licinfo.get('rev', 0) + token = licinfo.get('token', 0) + features = licinfo.get('features', 0) + if rev == 1 and features > 0 and token > 0: + logger.warning('this license is not ready for Pyarmor 9') + if silent: + return False + + # Group License + if features == 15: + prompt = ( + '', + 'Pyarmor 9 has some changes on license policy', + 'This group license is still available', + 'But it need request new device regfile as before', + 'Press "h" to check Pyarmor 9 Upgrade Notes', + '', + 'Help (h), Quit (q): ' + ) + show_help_page(prompt, URL_UPGRADE_V9) + raise SystemExit('Quit') + + prompt = ( + '', + 'Pyarmor 9 has big change on CI/CD pipeline', + 'If not using Pyarmor License in CI/CD pipeline', + 'Press "c" to continue', + 'Otherwise press "h" to check Pyarmor 9.0 Upgrade Notes', + '', + 'Continue (c), Help (h), Quit (q): ' + ) + if not show_help_page(prompt, URL_UPGRADE_V9) == 'c': + raise SystemExit('Quit') + + class Register(object): def __init__(self, ctx): @@ -101,6 +160,8 @@ def _get_old_rcode(self): def regurl(self, ucode, product=None, rcode=None, prepare=False): url = self.ctx.cfg['pyarmor']['regurl'] % ucode if product: + if len(product) > 60: + raise CliError('too long product name (length > 60)') url += '&product=' + \ urlsafe_b64encode(product.encode('utf-8')).decode() if rcode: @@ -110,7 +171,7 @@ def regurl(self, ucode, product=None, rcode=None, prepare=False): return url def update_token(self): - from .core import Pytransform3 + from pyarmor.cli.core import Pytransform3 assert Pytransform3._pytransform3 is None with open(self.ctx.license_token, 'wb') as f: f.close() @@ -121,9 +182,10 @@ def license_info(self): return parse_token(self.ctx.read_token()) def _license_type(self, info): - return 'basic' if info['features'] == 1 else \ + return 'basic' if info['features'] in (1, 17) else \ 'pro' if info['features'] == 7 else \ 'group' if info['features'] == 15 else \ + 'ci' if info['features'] == 23 else \ 'trial' if info['token'] == 0 else 'unknown' def _license_to(self, info): @@ -146,15 +208,97 @@ def parse_keyfile(self, filename): if len(line) == 192 and line.find(' ') == -1: return regname, line - raise CliError('no registration code found in %s' % filename) + logger.error('please make sure the activation file is ' + 'pure text file, the content starts with `Dear `, ' + 'and includes the activation code' + '(its length is 192 and no line break)') + raise CliError('invalid activation file "%s"' % filename) + + def _register_offline_license(self, fzip, namelist): + logger.info('machine id in group license: %s', ', '.join( + [x[7:] for x in namelist if x.startswith('tokens')])) + for idver in MACHFLAGS: + machid = self._get_machine_id(idver).decode('utf-8') + logger.debug('got machine id: %s', machid) + name = '/'.join(['tokens', machid]) + if name in namelist: + logger.info('this machine id matchs group license') + break + else: + logger.info('no machine id matchs this group license') + logger.info('take this machine as docker container, and ' + 'connect to docker host for authentication...') + mlist = self._get_docker_hostname() + if not mlist: + logger.info( + 'could not get docker host machine id\n%s', + '\n'.join([ + '', + 'if this machine is docker container, please ' + 'run command `pyarmor-auth` in docker host, ' + 'and try it again', '', + 'otherwise please generate new group ' + 'device license for this machine', '', + 'more information please check section ' + '"using group license" in documentation ' + '"how-to register" guide', '' + ])) + raise CliError('this group device license is not for ' + 'this machine') + for machid in mlist: + hostname = '/'.join(['tokens', machid]) + if hostname in namelist: + name = hostname + break + else: + logger.debug('docker host machine ids: %s', mlist) + raise CliError('this group device license is not for ' + 'this docker host') + + logger.debug('extracting %s', name) + self.ctx.save_group_token(fzip.read(name)) + + def _init_token(self, reginfo): + from struct import pack + rev, licno, lictp, regname, product = ( + reginfo['rev'], reginfo['rcode'], reginfo['type'], + reginfo['name'], reginfo['product'] + ) + + token = 0 + old_license = self.license_info + if old_license['licno'] == licno: + token = old_license['token'] + + features = (1 if lictp == 'J' else 7 if lictp == 'Z' else + 15 if lictp == 'G' else 23 if lictp == 'C' else + 0) + notes = ( + '* Do not use this file in CI/CD pipeline directly\n' + '* Only use it to request CI regfile ' + '"pyarmor-ci-%s.zip"' % licno[:6].lstrip('0') + if lictp == 'C' else '' + ) + + regname = regname.encode('utf-8') + product = product.encode('utf-8') + notes = notes.encode('utf-8') + sizes = len(regname), len(product), len(notes) + data = pack(' 0: - logger.info('last register was within %d seconds, ' - 'waiting for %d seconds', delta, d) + if activation: + logger.warning( + 'caution: this activation file "%s" can only ' + 'be used no more than 10 times', activation) + logger.info('waiting for %d seconds', d) while time() - st.st_mtime < delta: logger.info('waiting ...') sleep(3.0) @@ -436,16 +579,15 @@ def prepare(self, keyfile, product, upgrade=False): lines = [] if upgrade: if rcode and not rcode.startswith('pyarmor-vax-'): - logger.error('please check Pyarmor 8 EULA') raise CliError('old license "%s" can not be upgraded' % rcode) if info['upgrade']: lines.append(upgrade_to_pro_info.substitute(rcode=rcode)) else: lines.append(upgrade_to_basic_info.substitute()) else: - if info['lictype'] not in ('BASIC', 'PRO', 'GROUP'): - logger.error('this license does not work in Pyarmor 8') - logger.error('please check Pyarmor 8.0 EULA') + if info['lictype'] == 'OLD': + raise CliError('old license only works for Pyarmor <= 7.7.4') + if info['lictype'] not in ('BASIC', 'PRO', 'GROUP', 'CI'): raise CliError('unknown license type %s' % info['lictype']) lines.append('This license registration information will be') @@ -479,6 +621,8 @@ def prepare(self, keyfile, product, upgrade=False): return info, '\n'.join(lines) def upgrade_to_pro(self, keyfile, product): + raise CliError('this feature is not available now') + logger.info('process upgrading file "%s"', keyfile) reginfo = self.parse_keyfile(keyfile) @@ -490,7 +634,7 @@ def upgrade_to_pro(self, keyfile, product): logger.info('send upgrade request to server') res = self._send_request(url) - regfile = self._handle_response(res) + regfile, lictype = self._handle_response(res) logger.info('update license token') self.update_token() @@ -500,7 +644,7 @@ def upgrade_to_pro(self, keyfile, product): 'use this file for subsequent registration' % regfile, logger.info('Import Notes:\n\n%s\n', notes) - def register(self, keyfile, product, upgrade=False, group=False): + def register(self, keyfile, product=None, upgrade=False, group=False): if keyfile.endswith('.zip'): logger.info('register "%s"', keyfile) self.register_regfile(keyfile) @@ -510,63 +654,91 @@ def register(self, keyfile, product, upgrade=False, group=False): reginfo = self.parse_keyfile(keyfile) url = self.regurl(reginfo[1], product=product) + url += '&rev=' + str(self.LICENSE_REVSION) if upgrade: url += '&upgrade_to_basic=1' logger.debug('url: %s', url) logger.info('send request to server') res = self._send_request(url) - regfile = self._handle_response(res) + regfile, lictype = self._handle_response(res) + + logger.info('') + logger.info('the registration file "%s" has been generated', regfile) + logger.info('this license has been activated sucessfully') notes = [ - '* Please backup regfile "%s" carefully, and ' - 'use this file for subsequent registration' % regfile, - '* Do not use "%s" again' % os.path.basename(keyfile), + '* Please backup "%s", but do not use it to ' + 'register Pyarmor' % os.path.basename(keyfile), + '* Please backup regfile "%s", and ' + 'use this file for next any registration' % regfile, + '* Do not use this file in docker and CI/CD pipeline', + '', ] + logger.info('\n\nImport Notes:\n%s\n', '\n'.join(notes)) - if group: - logger.info('This group license has been activated sucessfully') - notes.append('* Please check `pyarmor reg` in Man page for ' - 'how to register Pyarmor on offline device') - else: - logger.info('register "%s"', regfile) - self.register_regfile(regfile) - logger.info('This license code has been %s successfully', - 'upgraded' if upgrade else 'activated') - - logger.info('Import Notes:\n\n%s\n', '\n'.join(notes)) + input('Type Enter to continue ...') + LicenseHelper(self).run(lictype, regfile) def _handle_response(self, res): if res and res.code == 200: + lictype = None dis = res.headers.get('Content-Disposition') filename = dis.split('"')[1] if dis else 'pyarmor-regfile.zip' logger.info('write registration file "%s"', filename) data = res.read() if data.startswith(b'{"group":'): + lictype = 'G' n = data.find(b'}') + 1 with open(filename, 'wb') as f: f.write(data[n:]) self._write_group_info(filename, data[:n]) + elif data.startswith(b'REGINFO:'): + i = len(b'REGINFO:') + n = data[i] + (data[i+1] << 8) + i += 2 + n += i + with open(filename, 'wb') as f: + f.write(data[n:]) + reginfo = data[i:n] + lictype = json_loads(reginfo).get('type', None) + self._write_reg_info(filename, reginfo) else: + # Only for request group token + logger.debug('no REGINFO found') with open(filename, 'wb') as f: f.write(data) - return filename + return filename, lictype elif res: - raise CliError(res.read().decode('utf-8')) + data = res.read() + logger.debug('server return(%d): %s', res.code, data) + try: + msg = data.decode('utf-8') + except Exception as e: + logger.debug('decode server data error "%s"', e) + msg = data + raise CliError(msg) raise CliError('no response from license server') + def _write_reg_info(self, filename, data): + from zipfile import ZipFile + logger.info('write reg information') + with ZipFile(filename, 'a') as f: + f.writestr('reg.info', data) + def _write_group_info(self, filename, data): from zipfile import ZipFile logger.info('write group information') with ZipFile(filename, 'a') as f: f.writestr('group.info', data) - def register_group_device(self, regfile, devid, rev=1): + def request_device_regfile(self, regfile, devid): from zipfile import ZipFile devfile = self.ctx.group_device_file(devid) - logger.info('register device file "%s"', devfile) + rev = self.LICENSE_REVSION + logger.info('request device regfile "%s" (v%d)', devfile, rev) logger.info('use group license "%s"', regfile) if not os.path.exists(devfile): logger.error('please generate device file in offline device by') @@ -610,7 +782,7 @@ def register_group_device(self, regfile, devid, rev=1): logger.debug('url: %s', url) res = self._send_request(url) - filename = self._handle_response(res) + filename, lictype = self._handle_response(res) with open(filename, 'rb') as f: data = f.read() os.makedirs(os.path.dirname(tokencache), exist_ok=True) @@ -621,7 +793,301 @@ def register_group_device(self, regfile, devid, rev=1): f.writestr('license.lic', licdata) f.writestr('.pyarmor_capsule.zip', capsule) f.writestr('group.tokens', b'') + f.writestr('dev.info', ('%s,%s' % (devid, machid)).encode('utf-8')) f.writestr('tokens/' + machid, data) logger.info('please copy deivce regfile to offline device and run') logger.info(' pyarmor reg %s', filename) + + register_group_device = request_device_regfile + + def _write_ci_info(self, filename, data): + from zipfile import ZipFile + logger.info('write ci information') + with ZipFile(filename, 'a') as f: + f.writestr('ci.token', data) + + def request_ci_regfile(self, regfile): + rev = self.LICENSE_REVSION + cirev = self.CI_LICENSE_REVSION + logger.info('request ci regfile (v%d) by "%s"', cirev, regfile) + from zipfile import ZipFile + + with ZipFile(regfile, 'r') as f: + if 'reg.info' not in f.namelist(): + logger.error('missing reg.info in regfile, ' + 'this license may be out of date, ' + 'please check Pyarmor upgrade notes') + raise CliError('can not request CI license') + reginfo = json_loads(f.read('reg.info')) + + ucode = reginfo['ucode'] + rcode = reginfo['rcode'] + if len(ucode) != 192: + raise CliError('invalid ci registration file "%s"', regfile) + + url = self.regurl('ci/%s' % ucode) + paras = ('rev', str(rev)), ('cirev', str(cirev)) + url += '&'.join(['='.join(x) for x in paras]) + logger.debug('url: %s', url) + + logger.info('send request to server') + res = self._send_request(url) + + logger.info('handle response') + if res is None: + raise CliError('no response from license server') + + elif res.code != 200: + raise CliError(res.read().decode('utf-8')) + + data = res.read() + if not data.startswith(b'CITOKEN:'): + raise CliError('wrong server data: "%s"' % data) + + i = len(b'CITOKEN:') + 2 + n = data[i-2] + (data[i-1] << 8) + token = data[i:i+n] + + rn = rcode[-6:].lstrip('0') + cifile = 'pyarmor-ci-%s.zip' % rn + + with ZipFile(regfile, 'r') as src: + with ZipFile(cifile, 'w') as dst: + for x in src.namelist(): + dst.writestr(x, src.read(x)) + dst.writestr('ci.token', token) + + logger.info('generate CI regfile "%s" successfully', cifile) + + ver = '.'.join([str(x) for x in self.ctx.version]) + logger.info('\n\nCheck CI license in local machine by:\n' + '\n\tpyarmor reg %s\n\n' + 'Register Pyarmor in CI/CD pipeline by:\n' + '\n\tpip install pyarmor==%s\n' + '\tpyarmor reg %s\n', + cifile, ver, cifile) + + +BASIC_LICENSE_HELP_INFO = Template(''' +Using Basic license in CI/CD pipeline or docker container need extra steps, please check this page + +$docurl/how-to/ci.html + +More usage about Basic License, check this page + +$docurl/how-to/register.html + +If need register Pyarmor in build device, run this command: + + pyarmor reg $regfile + +''') + +PRO_LICENSE_HELP_INFO = Template(''' +Pro license can't be used in CI/CD pipeline or docker container direclty. + +A few times for debug purpose, about 60 runs per month, may work. + +But there is one workaroud for Pro licnese in CI/CD pipeline, please check this page + +$docurl/how-to/ci.html + +More usage about Pro License, check this page + +$docurl/how-to/register.html + +If need register Pyarmor in build device, run this command: + + pyarmor reg $regfile + +''') + +GROUP_LICENSE_HELP_INFO = Template(''' +Group License file `$regfile` is only used to request device regfile + +In order to register Pyarmor in offline device: + +1. In the build device (may be offline), generate device info by this command + + pyarmor reg -g 1 + +2. In any online device, copy device file generated by first step, then request device regfile for this device + + cp pyarmor-group-device.1 .pyarmor/group/ + pyarmor reg -g 1 $regfile + +3. In the build device, register Pyarmor by device regfile + + pyarmor reg pyarmor-device-regfile-xxxx.1.zip + +More usage about CI License, check section `Using group licnese` + +$docurl/how-to/register.html +''') + +CI_LICENSE_HELP_INFO = Template(''' +CI license file `$regfile` is only used to request CI regfile + +CI regfile is used to register Pyarmor in the CI/CD pipeline + +If need request CI regfile, run this command: + + pyarmor reg -C $regfile + +Note that CI regfile can NOT be used in physical machine, and there is rate limits to register CI regfile in CI/CD pipeline and docker container. + +It may need request new CI regfile after Pyarmor is upgraded, please check section `When need to request new CI regfile` in this page + +$docurl/how-to/ci.html + +More usage about CI License, check these pages + +$docurl/how-to/register.html +$docurl/how-to/ci.html +''') + + +class LicenseHelper(object): + """Only used for first activate license. + + Help the beginner to understand how to use different licenses. + + """ + + def __init__(self, parent): + self.parent = parent + self.docurl = parent.ctx.cfg.get( + 'pyarmor', 'docurl').rstrip('/').replace('{lang}', 'en') + self.print = print + + def run(self, lictype, regfile): + if lictype == 'G': + self._group_license_helper(regfile) + + elif lictype == 'C': + self._ci_license_helper(regfile) + + elif lictype == 'Z': + self._pro_license_helper(regfile) + + elif lictype in ('J', 'B', 'P'): + self._basic_license_helper(regfile) + + else: + raise CliError('unknown license type "%s"' % lictype) + + def _basic_license_helper(self, regfile): + self.print(BASIC_LICENSE_HELP_INFO.substitute( + docurl=self.docurl, regfile=regfile)) + + prompt = 'Yes (y), No (n), Quit (q): ' + self.print('Show basic license usage in webbrowser? (n)') + choice = show_help_page([prompt], self.docurl + '/how-to/register.html') + if choice == 'q': + return + + self.print('Show basic license for CI/CD pipeline or docker? (n)') + choice = show_help_page([prompt], self.docurl + '/how-to/ci.html') + if choice == 'q': + return + + self.print('Do you want register Pyarmor in this machine? (y)') + choice = input(prompt).lower()[:1] + if choice == 'q': + return + + if choice in ('y', ''): + self.print('register "%s"' % regfile) + self.parent.register_regfile(regfile) + self.print('This license registration information:\n\n' + '%s' % self.parent) + + def _pro_license_helper(self, regfile): + self.print(PRO_LICENSE_HELP_INFO.substitute( + docurl=self.docurl, regfile=regfile)) + + prompt = 'Yes (y), No (n), Quit (q): ' + self.print('Show pro license usage in webbrowser? (n)') + choice = show_help_page([prompt], self.docurl + '/how-to/register.html') + if choice == 'q': + return + + self.print('Show pro license for CI/CD pipeline or docker? (n)') + choice = show_help_page([prompt], self.docurl + '/how-to/ci.html') + if choice == 'q': + return + + self.print('Do you want register Pyarmor in this machine? (y)') + choice = input(prompt).lower()[:1] + if choice == 'q': + return + + if choice in ('y', ''): + self.print('register "%s"' % regfile) + self.parent.register_regfile(regfile) + self.print('This license registration information:\n\n' + '%s' % self.parent) + + def _group_license_helper(self, regfile): + self.print(GROUP_LICENSE_HELP_INFO.substitute( + docurl=self.docurl, regfile=regfile)) + + prompt = 'Yes (y), No (n), Quit (q): ' + self.print('Show group license usage in webbrowser? (n)') + choice = show_help_page([prompt], self.docurl + '/how-to/register.html') + if choice == 'q': + return + + self.print('Do you want register Pyarmor in this machine? (n)') + choice = input(prompt).lower()[:1] + if choice == 'q': + return + + if choice != 'y': + return + + self.print('Please assign one unused device no. to this device, ' + 'starts from 1') + devid = None + while devid is None: + a = input('Type device no. (default is 1): ') + if a == '': + devid = 1 + elif (not a.isdigit()) or int(a) < 1 or int(a) > 100: + self.print('invalid input') + else: + devid = int(a) + + if devid: + sep = '-' * 16 + devinfo = self.parent.ctx.group_device_file(devid) + self.print('%s 1. generate device info' % sep) + if os.path.exists(devinfo): + logger.warning('old device file has been exists') + else: + self.parent.generate_group_device(devid) + self.print('%s 2. request device regfile' % sep) + self.parent.request_device_regfile(regfile, devid) + self.print('%s 3. register Pyarmor with device regfile' % sep) + self.parent.register(regfile. + replace('.zip', '.%d.zip' % devid). + replace('-regfile', '-device-regfile')) + + def _ci_license_helper(self, regfile): + self.print(CI_LICENSE_HELP_INFO.substitute( + docurl=self.docurl, regfile=regfile)) + + prompt = 'Yes (y), No (n), Quit (q): ' + self.print('Show ci license usage in webbrowser? (n)') + choice = show_help_page([prompt], self.docurl + '/how-to/ci.html') + if choice == 'q': + return + + self.print('Do you want request one CI Regfile now? (y)') + choice = input(prompt).lower()[:1] + if choice == 'q': + return + + if choice in ('y', ''): + self.parent.request_ci_regfile(regfile) diff --git a/src/cli/repack.py b/src/cli/repack.py index 5a4db2a5..5338e716 100644 --- a/src/cli/repack.py +++ b/src/cli/repack.py @@ -96,7 +96,7 @@ def autoclean_output(output, autoclean=True): with open(resfile, 'wb') as f: marshal.dump(mlist + list(plist), f) -with open(hookscript, 'w') as f: +with open(hookscript, 'w', encoding='utf-8') as f: f.write("hiddenimports=[%s]" % ", ".join([repr(x) for x in hiddenimports])) ''' diff --git a/src/cli/resource.py b/src/cli/resource.py index fade2cb8..249f6b29 100644 --- a/src/cli/resource.py +++ b/src/cli/resource.py @@ -109,22 +109,34 @@ def is_pyc(self): return self.pyext.lower() == '.pyc' def _get_encoding(self, encoding): + # Maybe: from tokenize import detect_encoding from codecs import BOM_UTF8 from re import search as research + + def find_encoding(line): + try: + pat = r'coding[=:]\s*([-\w.]+)' + n = line.find(b'\n') + if n > -1: + m = research(pat, line[:n].decode()) + if m: + return m.group(1) + if line[n+1] == 35: + k = n + 1 + n = line.find(b'\n', k) + m = research(pat, line[k:n].decode()) + if m: + return m.group(1) + except Exception: + # Ignore decoding error or index error + pass + with open(self.fullpath, 'rb') as f: line = f.read(80) if line and line[:3] == BOM_UTF8: return 'utf-8' if line and line[0] == 35: - n = line.find(b'\n') - m = research(r'coding[=:]\s*([-\w.]+)', line[:n].decode()) - if m: - return m.group(1) - if n > -1 and len(line) > (n+1) and line[n+1] == 35: - k = n + 1 - n = line.find(b'\n', k) - m = research(r'coding[=:]\s*([-\w.]+)', line[k:n].decode()) - return m and m.group(1) + encoding = find_encoding(line) or encoding return encoding def readlines(self, encoding=None): diff --git a/src/cli/shell.py b/src/cli/shell.py index 62b1954c..8325b0cc 100644 --- a/src/cli/shell.py +++ b/src/cli/shell.py @@ -3,12 +3,12 @@ # ############################################################# # # -# Copyright @ 2023 - Dashingsoft corp. # +# Copyright @ 2024 - Dashingsoft corp. # # All rights reserved. # # # # Pyarmor # # # -# Version: 8.0.1 - # +# Version: 9.1.0 - # # # ############################################################# # @@ -17,65 +17,465 @@ # # @Author: Jondy Zhao (pyarmor@163.com) # -# @Create Date: Thu Jan 12 10:27:05 CST 2023 +# @Create Date: Fri Nov 15 10:52:00 CST 2024 # import configparser import cmd -import os +import shlex +from .model import SECTIONS, GROUPS -class PyarmorShell(cmd.Cmd): - intro = 'Welcome to the Pyarmor shell. Type help or ? to list commands.\n' - prompt = '(pyarmor) ' +class CfgEntity: + """Store option to cfg file""" - def __init__(self, ctx): - super().__init__() - self.ctx = ctx - self._reset() + def __init__(self, cfgfiles, encoding=None): + self._cfgfiles = cfgfiles + self._reader = None + self._writer = None + self._index = -1 + self._encoding = encoding - def _reset(self): - ctx = self.ctx + def _load_cfg(self, filenames): cfg = configparser.ConfigParser( empty_lines_in_values=False, interpolation=configparser.ExtendedInterpolation(), ) - cfg.read([ctx.default_config, ctx.global_config, ctx.local_config]) - self._cfg = cfg + cfg.read(filenames, encoding=self._encoding) + return cfg + + @property + def cfgfile(self): + return self._cfgfiles[self._index] + + @property + def reader(self): + if self._reader is None: + self._reader = self._load_cfg(self._cfgfiles) + return self._reader + + @property + def writer(self): + if self._writer is None: + self._writer = self._load_cfg(self.cfgfile) + return self._writer + + def store(self, sect, opt, value): + for cfg in (self.writer, self.reader): + if not cfg.has_section(sect): + cfg.add_section(sect) + cfg.set(sect, opt, value) + + def fetch(self, sect, opt): + cfg = self.reader + if cfg.has_section(sect): + return cfg[sect].get(opt, None) + + def remove(self, sect, opt=None): + for cfg in (self.writer, self.reader): + if cfg.has_section(sect): + if opt is None: + cfg.remove_section(sect) + else: + cfg.remove_option(sect, opt) + + def reset(self, index): + if self._index == index: + return + self._index = index + self._reader = self._writer = None + + def save(self): + with open(self.cfgfile, 'w', encoding=self._encoding) as f: + self.writer.write(f) + + +class Domain: + + def __init__(self, name, default='pyarmor'): + self.name = name + self._dirty = False + + self._sections = [] + self._groups = [] + + self._default = Section(default, parent=self) + self._cfgentity = None + self._children = None + + @property + def qualname(self): + return self.name + + @property + def title(self): + return '(%s) ' % self.name + + @property + def dirty(self): + return self._dirty + + @dirty.setter + def dirty(self, value): + self._dirty = value + + @property + def groups(self): + for x in self._groups: + yield x + + @property + def sections(self): + for x in self._sections: + yield x + + @property + def options(self): + if self._default: + for x in self._default.options: + yield x + + @property + def children(self): + if self._children is None: + items = {x.name: x for x in self.groups} + items.update({x.name: x for x in self.sections}) + self._children = items + return self._children + + @property + def cfgentity(self): + return self._cfgentity + + def link(self, cfgentity): + self._cfgentity = cfgentity + + def save(self, force=False): + cfgentity = self._cfgentity + if (self.dirty or force) and cfgentity: + cfgentity.save() + self._dirty = False + + def add_section(self, name): + if self._default and self._default.name == name: + return + self._sections.append(Section(name, parent=self)) + + def add_group(self, name, sections): + group = Group(name, parent=self) + group.add_sections(sections) + self._groups.append(group) + + +class Section: + + def __init__(self, name, parent=None): + self.parent = parent + self.name = name + self.models = SECTIONS.get(name, []) + self._options = None + self._children = {} + + @property + def qualname(self): + parent = self.parent + name = self.name + return ':'.join([parent.qualname, name]) + + @property + def title(self): + return '({0})[{1}] '.format(*self.qualname.split(':', 1)) + + @property + def options(self): + if self._options is None: + self._options = [x.factory(self) for x in self.models] + + for x in self._options: + yield x + + @property + def children(self): + return self._children - def _reset_prompt(self): - prompts = ['(pyarmor) '] - self.prompt = '\n'.join(prompts) + @property + def cfgentity(self): + return self.parent.cfgentity - def do_exit(self, arg): - 'Finish config and exit' - print('Thank you for using Pyarmor') + def store(self, opt, value): + self.cfgentity.store(self.name, opt, value) + + def fetch(self, opt): + return self.cfgentity.fetch(self.name, opt) + + def remove(self, opt): + self.cfgentity.remove(self.name, opt) + + +class Group(Section): + + def __init__(self, name, parent=None, sections=None): + super().__init__(name, parent=parent) + self._groups = [] + self._sections = [] + + @property + def group(self): + for x in self._group: + yield x + + @property + def sections(self): + for x in self._sections: + yield x + + @property + def children(self): + if self._children is None: + items = {x.name: x for x in self.groups} + items.update({x.name: x for x in self.sections}) + self._children = items + return self._children + + def add_group(self, name, sections): + group = Group(name, parent=self) + group.add_sections(sections) + self._group.append(group) + + def add_sections(self, sections): + self._sections.extend([ + Section(x, parent=self) for x in sections + ]) + + +class PyarmorShell(cmd.Cmd): + + intro = 'Type help or ? to list commands.\n' + col = 60 + + def __init__(self, ctx, domain='local'): + super().__init__() + self.ctx = ctx + self._entity = CfgEntity([ + ctx.default_config, + ctx.global_config, + ctx.local_config + ], encoding=ctx.encoding) + self._domains = self._init_domains() + self._domain = self._section = self._domains[domain] + + def _init_domains(self): + gs = Domain('global') + ls = Domain('local') + ps = Domain('project', default='project') + + gs.link(self._entity) + ls.link(self._entity) + ps.link(self._entity) + + sections = 'finder', 'builder', 'runtime', 'pack' + [(gs.add_section(x), ls.add_section(x)) for x in sections] + + for domain in (gs, ls): + for name in ('filter', 'bcc'): + domain.add_group(name, GROUPS[name]) + + ps.add_section('rft') + ps.add_section('mini') + ps.add_section('vmc') + ps.add_section('ecc') + + return {'global': gs, 'local': ls, 'project': ps} + + @property + def section(self): + return self._section + + @property + def prompt(self): + return self.section.title + + def get_sections(self, name=''): + obj = self.section + sects = [x for x in getattr(obj, 'groups', [])] + sects.extend([x for x in getattr(obj, 'sections', [])]) + return (sects if name in ('-a', '*', '') else + [x for x in sects if x.name.startswith(name)]) + + def get_options(self, name=''): + options = [x for x in self.section.options] + return (options if name in ('-a', '*', '') else + [x for x in options if x.name.startswith(name)]) + + def printf(self, *args): + print(*args) + + def find(self, opt): + for item in self.section.options: + if item.name == opt: + return item + + def save(self): + self._domain.save() + + def do_exit(self, arg=None): + """Finish config and exit""" + self.save() + self.printf('') + self.printf('Thank you for using Pyarmor') return True do_EOF = do_q = do_exit def do_use(self, arg): - 'Select config file' + """Switch domain: global, local, project""" + names = list(self._domains.keys()) + if arg not in names: + self.printf('Invalid name: ', arg) + self.printf('Please type one of', str(names)) + elif self._domain.name == arg: + pass + else: + self._domain.save() + if arg == 'global': + self._entity.reset(index=1) + elif self._domain.name == 'global': + self._entity.reset(index=-1) + self._domain = self._domains[arg] + self._section = self._domain def do_ls(self, arg): - '''List all the available items in current scope''' + """List all the available items in current domain""" + sections = self.get_sections(arg) + if sections: + self.printf('Sections:') + self.columnize([x.name for x in sections], self.col) + self.printf('') + + options = self.get_options(arg) + if options: + self.printf('Options:') + self.columnize([x.name for x in options], self.col) + self.printf('') def do_cd(self, arg): - '''Change scope''' + """Switch to section, .. to parent, blank to top""" + if arg == '..': + if self._section is not self._domain: + self._section = self._section.parent + elif arg: + item = self.section.children.get(arg) + if item: + self._section = item + else: + self.printf('No item', repr(arg)) + else: + self._section = self._domain - def do_rm(self, arg): - '''Remove item in the scope''' + def do_get(self, arg): + """Show option value""" + options = self.get_options(arg) + if not options: + self.printf('No found any', repr(arg)) + return + for opt in options: + value = opt.value + if value is not None: + self.printf('%-20s= %s' % (opt.name, value)) + + def do_info(self, arg): + """List sections, options, and all the values""" + if arg == '': + self.do_ls('') + self.do_get('') + return + item = self.section.children.get(arg) + if item: + self.do_cd(arg) + self.do_ls('') + self.do_get('') + self.do_cd('..') + else: + item = self.find(arg) + if item: + self.printf(item.usage()) + self.do_get(arg) + else: + self.printf('No found', arg) def do_set(self, arg): - 'Change option value' + """Set option value""" + if arg.find(' ') == -1: + self.printf('Missing value') + return + name, value = arg.split(None, 1) + item = self.find(name) + if item: + c = value[:1] + if c in ('"', "'"): + value = value.strip(c) + try: + item.value = value + self._domain.dirty = True + except ValueError as e: + self.printf(str(e)) + else: + self.printf('No found', name) - def do_show(self, arg): - 'Show option value' + def do_reset(self, arg): + """Reset option in the domain""" + if arg in ('', '*'): + self.printf('Please specify one option') + return + item = self.find(arg) + if item: + item.reset() + self._domain.dirty = True + else: + self.printf('No found', arg) + def do_push(self, arg, silent=False): + """Append new value to option""" + if arg.find(' ') == -1: + self.printf('Missing value') + return + name, value = arg.split(None, 1) + item = self.find(name) + if item: + for x in shlex.split(value): + try: + item.push(x) + except ValueError as e: + self.printf(str(e)) + break + else: + self._domain.dirty = True + self.do_get(name) + else: + self.printf('No found', name) -def parse(arg): - 'Convert a series of zero or more numbers to an argument tuple' - return tuple(map(int, arg.split())) + def do_pop(self, arg): + """Remove one choice from option""" + if arg.find(' ') == -1: + self.printf('Missing value') + return + name, value = arg.split(None, 1) + item = self.find(name) + if item: + for x in shlex.split(value): + item.pop(x) + self._domain.dirty = True + self.do_get(name) + else: + self.printf('No found', name) if __name__ == '__main__': - PyarmorShell().cmdloop() + from os.path import join as joinpath, abspath, expanduser + + from .context import Context + + home = joinpath('~', '.pyarmor') + home = abspath(expanduser(home)) + + ctx = Context(home) + PyarmorShell(ctx).cmdloop() diff --git a/src/config.py b/src/config.py index d594369d..03e63294 100755 --- a/src/config.py +++ b/src/config.py @@ -1,6 +1,6 @@ from sys import platform -version = '8.5.12' +version = '9.2.7' # The corresponding version of pytransform.so core_version = 'r52.6' @@ -43,7 +43,7 @@ key_url = 'https://api.dashingsoft.com/product/key/%s/query' reg_url = 'https://api.dashingsoft.com/product/key/activate/%s/' -buy_url = 'https://order.shareit.com/cart/add?vendorid=200089125&PRODUCT[300871197]=1' +buy_url = 'https://jondy.github.io/paypal/index.html' help_url = 'https://pyarmor.readthedocs.io/{lang}/v%s/{page}' % version sppmode_info = { diff --git a/tests.8/Makefile b/tests.8/Makefile index e0e3d017..6fb7913b 100644 --- a/tests.8/Makefile +++ b/tests.8/Makefile @@ -3,31 +3,38 @@ HOSTS := darwin.x86_64 darwin.arm64 linux.x86_64 linux.aarch64 linux.x86 windows UNAME = $(shell uname) ARCH := $(shell arch) +ALL_PYVERS := 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 + ifeq ($(findstring Linux,$(UNAME)),Linux) ifeq ($(ARCH),x86_64) HOST ?= linux.x86_64 - PYVERS := 3.7 3.8 3.9 3.10 3.11 3.12 + PYVERS := ${ALL_PYVERS} PYTHONBIN := /usr/local/bin/pythonXY else ifeq ($(ARCH),aarch64) HOST ?= linux.aarch64 - PYVERS := 3.10 - PYTHONBIN := /usr/local/bin/pythonXY + PYVERS := 3.9 3.10 3.11 3.12 3.13 3.14 3.15 + PYTHONBIN := /usr/bin/pythonXY else HOST ?= linux.x86 - PYVERS := 3.7 3.8 3.9 3.10 3.11 3.12 + PYVERS := ${ALL_PYVERS} PYTHONBIN := /usr/local/bin/pythonXY endif else ifeq ($(findstring Darwin,$(UNAME)),Darwin) ifeq ($(ARCH),arm64) HOST ?= darwin.arm64 - PYVERS := 3.9 3.10 - PYTHONBIN := /usr/local/bin/pythonXY + PYVERS := 3.9 3.10 3.11 3.12 3.13 3.14 3.15 + PYTHONBIN := /Users/zhaojunde/workspace/v9-test/venv/XY/bin/pythonXY else HOST ?= darwin.x86_64 - PYVERS := 3.7 3.8 3.9 3.10 3.11 3.12 + PYVERS := ${ALL_PYVERS} PYTHONBIN := /usr/local/bin/pythonXY endif -else +else ifeq (${HOSTNAME},584VVL9L9XIZV8H) + # This is win10 test server + HOST ?=windows.x86_64 + PYVERS := 3.9 3.10 3.11 3.12 3.13 3.14 + PYTHONBIN := ~/v9-test/venv/XY/Scripts/python.exe +else ifeq (${HOSTNAME},WIN7-1611171239) ifeq (${HOST},windows.x86) PYVERS := 3.7 3.8 PYTHONBIN := C:/PythonXY-32/python @@ -51,7 +58,9 @@ ifneq ($(findstring ${PYVER},${PYVERS}),${PYVER}) $(error "${PYVER}" not in supported Python versions "${PYVERS}") endif -ifeq (${HOST},windows.x86_64) +ifeq (${HOSTNAME},584VVL9L9XIZV8H) + PYTHON ?= $(subst XY,${PYVER},${PYTHONBIN}) +else ifeq (${HOST},windows.x86_64) PYTHON ?= $(subst XY,$(subst .,,${PYVER}),${PYTHONBIN}) else ifeq (${HOST},windows.x86) PYTHON ?= $(subst XY,$(subst .,,${PYVER}),${PYTHONBIN}) @@ -81,8 +90,8 @@ whlpath := $(shell (test -d ${whlpath} && echo ${whlpath}) || echo ./pkgdist) pkgfile := $(word 1,$(shell cd ${pkgpath}; ls -t pyarmor-*.tar.gz)) testver := $(subst pyarmor-,,$(subst .tar.gz,,${pkgfile})) -coreprefix := pyarmor.cli.core- -coresuffix := -cp38-none-macosx_10_14_x86_64.whl +coreprefix := pyarmor_cli_core- +coresuffix := -cp38-cp38-macosx_10_14_x86_64.whl corever=$(subst ${coreprefix},,$(subst ${coresuffix},,$(shell cd ${whlpath}; ls -t ${coreprefix}*${coresuffix}))) $(info got test package file: ${pkgfile}) @@ -101,13 +110,13 @@ test-%: PYVER=$(subst test-,,$@) make install-$(subst test-,,$@) atest ptest install-%: - @rm -rf ./pyarmor ./pyarmor.cli.core.themida* + @rm -rf ./pyarmor ./pyarmor_cli_core_themida* @tar xzf ${pkgpath}/pyarmor-${testver}.tar.gz @mv pyarmor-${testver}/pyarmor . @rm -rf pyarmor-${testver} - @unzip ${whlpath}/pyarmor.cli.core-${corever}-cp$(subst .,,$(subst install-,,$@))-none-${platform}.whl > /dev/null - @(themida=../pyarmor.cli.core.themida-${corever}-cp$(subst .,,$(subst install-,,$@))-none-any.whl && test -f $$themida && unzip $$themida > /dev/null || echo no themida $$themida) - @rm -rf pyarmor.cli.core-${corever}.dist-info + @unzip ${whlpath}/pyarmor_cli_core-${corever}-cp$(subst .,,$(subst install-,,$@))-cp$(subst .,,$(subst install-,,$@))-${platform}.whl > /dev/null + @(themida=../pyarmor_cli_core_themida-${corever}-cp$(subst .,,$(subst install-,,$@))-cp$(subst .,,$(subst install-,,$@))-any.whl && test -f $$themida && unzip $$themida > /dev/null || echo no themida $$themida) + @rm -rf pyarmor_cli_core-${corever}.dist-info atest: ${PYTHON} accept_test.py @@ -117,3 +126,7 @@ ptest: clean: rm -rf ${homepath} ${localpath} ./pyarmor ./pyarmor-* ./pyarmor.*.log + +# Unused +install-mini: + -@test -d ${pkgpath}/mini.extensions/${HOST} && mkdir -p pyarmor/mini/ && cp ${pkgpath}/mini.extensions/${HOST}/libs/cp${PYVER}/pyarmor_mini.* ./pyarmor/mini/ diff --git a/tests.8/accept_test.py b/tests.8/accept_test.py index a3a0191d..13e183ff 100755 --- a/tests.8/accept_test.py +++ b/tests.8/accept_test.py @@ -3,7 +3,6 @@ import logging import os import shutil -import sys import time import unittest @@ -11,6 +10,9 @@ from test.support import script_helper +is_win7 = os.getenv('HOSTNAME', '').startswith('WIN7-') + + def metricmethod(func): if not hasattr(time, 'process_time'): time.process_time = time.clock @@ -227,13 +229,13 @@ def test_bind_multiple_devices(self): self.pyarmor_gen(args) self.verify_dist_foo() - @unittest.skipUnless(sys.platform.startswith('win'), 'only for windows') + @unittest.skipUnless(is_win7, 'only for windows 7') def test_themida_script(self): args = ['g', '--enable-themida', 'samples/foo.py'] self.pyarmor_gen(args) self.verify_dist_foo() - @unittest.skipUnless(sys.platform.startswith('win'), 'only for windows') + @unittest.skipUnless(is_win7, 'only for windows 7') def test_themida_bcc(self): args = ['g', '--enable-themida', '--enable-bcc', 'samples/foo.py'] self.pyarmor_gen(args) @@ -313,7 +315,7 @@ def test_bcc_filter(self): ]) args = ['g', '--enable-bcc', 'samples/queens.py'] self.pyarmor_gen(args) - with open(os.path.join(self.local_path, 'pyarmor.trace.log')) as f: + with open('pyarmor.trace.log') as f: output = f.read() for line in ( 'trace.bcc ! queens:30:Queens.solve (excluded)', @@ -329,7 +331,7 @@ def test_rft_bcc_filter(self): ]) args = ['g', '--enable-bcc', '--enable-rft', 'samples/queens.py'] self.pyarmor_gen(args) - with open(os.path.join(self.local_path, 'pyarmor.trace.log')) as f: + with open('pyarmor.trace.log') as f: output = f.read() for line in ( 'trace.bcc ! queens:30', @@ -362,6 +364,36 @@ def test_mp_with_bcc(self): 'hello bob', ]])) + def test_thread(self): + args = ['g', 'samples/td.py'] + self.pyarmor_gen(args) + rc, stdout, stderr = self.assert_python_ok('dist/td.py') + lines = [x.strip() for x in stdout.splitlines()] + self.assertTrue(all([x.encode() in lines for x in [ + 'module name: __main__', + 'function f', + 'hello bob', + ]])) + + @only_protest + def test_thread_with_bcc(self): + args = ['g', '--enable-bcc', 'samples/td.py'] + self.pyarmor_gen(args) + rc, stdout, stderr = self.assert_python_ok('dist/td.py') + lines = [x.strip() for x in stdout.splitlines()] + self.assertTrue(all([x.encode() in lines for x in [ + 'module name: __main__', + 'function f', + 'hello bob', + ]])) + + @only_protest + def test_bcc_with_nest_functions(self): + args = ['g', '--enable-bcc', 'samples/nest.py'] + self.pyarmor_gen(args) + rc, stdout, stderr = self.assert_python_ok('dist/nest.py') + self.assertIn(b'test nest OK', stdout) + if __name__ == '__main__': logging.getLogger().addHandler(logging.NullHandler()) diff --git a/tests.8/samples/mp.py b/tests.8/samples/mp.py new file mode 100644 index 00000000..c232d0ba --- /dev/null +++ b/tests.8/samples/mp.py @@ -0,0 +1,18 @@ +from multiprocessing import Process +import os + +def info(title): + print(title) + print('module name:', __name__) + print('parent process:', os.getppid()) + print('process id:', os.getpid()) + +def f(name): + info('function f') + print('hello', name) + +if __name__ == '__main__': + info('main line') + p = Process(target=f, args=('bob',)) + p.start() + p.join() diff --git a/tests.8/samples/nest.py b/tests.8/samples/nest.py new file mode 100644 index 00000000..e9a739df --- /dev/null +++ b/tests.8/samples/nest.py @@ -0,0 +1,17 @@ +def outer(): + msg = 'this is inner2' + + def inner(): + return 'inner-result' + + def inner2(): + return msg + + print(inner2()) + return inner() + + +if __name__ == '__main__': + print('calling outer()...') + print('outer() returned:', outer()) + print('test nest OK') diff --git a/tests.8/samples/td.py b/tests.8/samples/td.py new file mode 100644 index 00000000..2450ebd3 --- /dev/null +++ b/tests.8/samples/td.py @@ -0,0 +1,21 @@ +from threading import Thread + + +def info(title): + print(title) + print('module name:', __name__) + + +def f(name): + info('function f') + print('hello', name) + + +if __name__ == '__main__': + tlist = [] + for i in range(10): + p = Thread(target=f, args=('bob',)) + p.start() + tlist.append(p) + for p in tlist: + p.join() diff --git a/tests.9/Makefile b/tests.9/Makefile new file mode 100644 index 00000000..f928a912 --- /dev/null +++ b/tests.9/Makefile @@ -0,0 +1,150 @@ +# +# Run tests in different platforms and different Python +# +# 1. Basic usage, test all pythons in this platform +# +# make test +# make nogil-test +# +# 2. Test one python version +# +# make test-3.13 +# +# 3. Test free-threaded version +# +# make nogil-test-3.13 +# +# Prerequisites: +# +# - Python has been installed in the path "venv" +# - Pyarmor has been registerred (Pro features) +# +HOSTS := darwin.x86_64 darwin.arm64 linux.x86_64 linux.aarch64 windows.x86_64 + +UNAME ?= $(shell uname) +ARCH ?= $(shell arch) +PYVERS ?= 3.9 3.10 3.11 3.12 3.13 3.14 +PYVERS_T ?= 3.13 3.14 + +PYVER ?= $(word 1,${PYVERS}) +ifneq ($(findstring ${PYVER},${PYVERS}),${PYVER}) + $(error "${PYVER}" not in supported Python versions "${PYVERS}") +endif + +ifeq ($(findstring Linux,$(UNAME)),Linux) + ifeq ($(ARCH),x86_64) + HOST ?= linux.x86_64 + PYTHON := /usr/local/bin/python${PYVER} + else ifeq ($(ARCH),aarch64) + HOST ?= linux.aarch64 + PYTHON := /usr/bin/python${PYVER} + else + HOST ?= linux.x86 + PYTHON := /usr/local/bin/python${PYVER} + endif + + PYTHON_T := ${PYTHON}t + +else ifeq ($(findstring Darwin,$(UNAME)),Darwin) + ifeq ($(ARCH),arm64) + HOST ?= darwin.arm64 + PYTHON := venv/${PYVER}/bin/python${PYVER} + PYTHON_T := venv/${PYVER}t/bin/python${PYVER} + else + HOST ?= darwin.x86_64 + PYTHON := venv/${PYVER}/bin/python${PYVER} + PYTHON_T := venv/${PYVER}t/bin/python${PYVER} + endif + +else + ifeq ($(ARCH),x86_64) + HOST ?=windows.x86_64 + PYTHON := venv/${PYVER}/Scripts/python.exe + PYTHON_T := venv/${PYVER}t/Scripts/python.exe + else + HOST ?=windows.x86 + PYTHON := venv/${PYVER}/Scripts/python.exe + PYTHON_T := venv/${PYVER}t/Scripts/python.exe + endif +endif + +PIPINST = install --force-reinstall --disable-pip-version-check +ifeq (${HOST},linux.aarch64) + PIP := ${PYTHON} pip.pyz ${PIPINST} + PIP_T := ${PYTHON_T} pip.pyz ${PIPINST} +else + PIP := ${PYTHON} -m pip ${PIPINST} + PIP_T := ${PYTHON_T} -m pip ${PIPINST} +endif + +$(info python is ${PYTHON}) + +PLATFORM = $(strip $(subst linux.x86,manylinux1_i686,\ + $(subst linux.x86_64,manylinux1_x86_64,\ + $(subst linux.aarch64,manylinux2014_aarch64,\ + $(subst linux.armv7,manylinux2014_armv7l,\ + $(subst windows.x86,win32,\ + $(subst windows.x86_64,win_amd64,\ + $(subst darwin.arm64,macosx_11_0_arm64,\ + $(subst darwin.x86_64,macosx_10_14_x86_64,\ + ${HOST}))))))))) + +PYTAG=cp$(subst .,,${PYVER}) + +workpath := __runner__ +pkgpath := ${workpath}/pkgdist +testpkg := $(wildcard ${pkgpath}/pyarmor_cli-*-none-any.whl) +corepkg := $(wildcard ${pkgpath}/pyarmor_cli_core-*-${PYTAG}-*-${PLATFORM}.whl) +minipkg := $(wildcard ${pkgpath}/pyarmor_mini-*-${PYTAG}-${PYTAG}-${PLATFORM}.whl) +minipkg_t := $(wildcard ${pkgpath}/pyarmor_mini-*-${PYTAG}-${PYTAG}t-${PLATFORM}.whl) + +ifndef testpkg +$(error no test package found) +endif + +ifndef corepkg +$(error no core package found) +endif + +ifndef minipkg +$(error no mini package found) +endif + +$(info got test package: $(notdir ${testpkg})) +$(info got core package: $(notdir ${corepkg})) +$(info got mini package: $(notdir ${minipkg})) + +.PHONY: all test nogil-test test-* notil-test-* install clean + +all: test nogil-test + +test: + for ver in ${PYVERS} ; do \ + make test-$$ver ; \ + done + +nogil-test: + for ver in ${PYVERS_T} ; do \ + make nogil-test-$$ver ; \ + done + +install: + ${PIP} ${corepkg} ${testpkg} ${minipkg} + +test-%: + PYVER=$(subst test-,,$@) make install run-test + +nogil-install: + ${PIP_T} ${minipkg_t} + +nogil-test-%: + PYVER=$(subst nogil-test-,,$@) make install nogil-install run-nogil-test + +run-test: + ${PYTHON} ${workpath}/accept_test.py + +run-nogil-test: + PYTHON_NOGIL=${PYTHON_T} ${PYTHON} ${workpath}/accept_test_nogil.py + +clean: + rm -rf ${workpath} __pycache__ testv9.tar.gz diff --git a/tests.9/README.md b/tests.9/README.md new file mode 100644 index 00000000..ba654e10 --- /dev/null +++ b/tests.9/README.md @@ -0,0 +1,7 @@ +# V9 Tests + +Test project, mini, rft, vmc, ecc scripts which are new features in Pyarmor 9+ + +Make sure package `pyarmor.cli>=9.2.2` and `pyarmor.mini>=3.0` has been installed: + + python accept_test.py diff --git a/tests.9/accept_test.py b/tests.9/accept_test.py new file mode 100755 index 00000000..0698e8fb --- /dev/null +++ b/tests.9/accept_test.py @@ -0,0 +1,132 @@ +# -*- coding: utf-8 -*- + +import logging +import os +import shutil +import unittest + +from test.support import script_helper + +from script_factory import script_generator + + +class BaseTestCase(unittest.TestCase): + + work_path = os.path.abspath(os.path.dirname(__file__)) + + @classmethod + def setUpClass(cls): + cls.local_path = os.path.join(cls.work_path, '.pyarmor') + cls.default_output = os.path.join(cls.work_path, 'dist') + + def setUp(self): + shutil.rmtree(self.local_path, ignore_errors=True) + + def tearDown(self): + pass + + def assert_python_ok(self, *args): + kwargs = { + '__isolated': False + } + return script_helper.assert_python_ok(*args, **kwargs) + + def assert_python_failure(self, *args): + kwargs = { + '__isolated': False + } + return script_helper.assert_python_failure(*args, **kwargs) + + def verify_script_pass(self, script, expected): + rc, stdout, stderr = self.assert_python_ok(script) + self.assertEqual(expected, (rc, stdout, stderr)) + + def build_v9_script(self, script, target): + kwargs = { + '__isolated': False, + '__cwd': self.work_path, + } + args = ['-m', 'pyarmor.cli', 'init', '-e', os.path.abspath(script)] + script_helper.assert_python_ok(*args, **kwargs) + + args = ['-m', 'pyarmor.cli', 'build', '--' + target] + script_helper.assert_python_ok(*args, **kwargs) + return os.path.join(self.default_output, os.path.basename(script)) + + def _test_target(self, target, catalog=None): + for script in self.iter_scripts(catalog): + with self.subTest(script=script): + expected = self.assert_python_ok(script) + obfscript = self.build_v9_script(script, target) + self.verify_script_pass(obfscript, expected) + + def make_script(self, source, name='foo'): + return script_helper.make_script(self.work_path, name, source) + + def iter_samples(self): + path = os.path.join(self.work_path, 'samples') + with os.scandir(path) as it: + for entry in it: + if entry.name.endswith('.py') and entry.is_file(): + yield entry.path + + def iter_scripts(self, catalog): + if catalog: + for name, source in script_generator(catalog): + yield self.make_script(source, name=name) + else: + yield from self.iter_samples() + + +class UnitTestCases(BaseTestCase): + + def test_mini_mode(self): + self._test_target('mini') + self._test_target('mini', 'modules') + self._test_target('mini', 'functions') + self._test_target('mini', 'methods') + + def test_vmc_mode(self): + self._test_target('vmc') + self._test_target('vmc', 'modules') + self._test_target('vmc', 'functions') + self._test_target('vmc', 'methods') + + def test_ecc_mode(self): + self._test_target('ecc') + self._test_target('ecc', 'modules') + self._test_target('ecc', 'functions') + self._test_target('ecc', 'methods') + + def test_rft_mode(self): + self._test_target('rft') + self._test_target('rft', 'modules') + self._test_target('rft', 'functions') + self._test_target('rft', 'methods') + + def test_mini_rft_mode(self): + self._test_target('mini-rft') + self._test_target('mini-rft', 'modules') + self._test_target('mini-rft', 'functions') + self._test_target('mini-rft', 'methods') + + def test_vmc_rft_mode(self): + self._test_target('vmc-rft') + self._test_target('vmc-rft', 'modules') + self._test_target('vmc-rft', 'functions') + self._test_target('vmc-rft', 'methods') + + def test_ecc_rft_mode(self): + self._test_target('ecc-rft') + self._test_target('ecc-rft', 'modules') + self._test_target('ecc-rft', 'functions') + self._test_target('ecc-rft', 'methods') + + +if __name__ == '__main__': + logging.getLogger().addHandler(logging.NullHandler()) + + loader = unittest.TestLoader() + # loader.testMethodPrefix = 'test_exclude_' + suite = loader.loadTestsFromTestCase(UnitTestCases) + result = unittest.TextTestRunner(verbosity=2).run(suite) diff --git a/tests.9/accept_test_nogil.py b/tests.9/accept_test_nogil.py new file mode 100755 index 00000000..8dd7003c --- /dev/null +++ b/tests.9/accept_test_nogil.py @@ -0,0 +1,135 @@ +# -*- coding: utf-8 -*- + +import logging +import os +import shutil +import unittest + +from test.support import script_helper + +from script_factory import script_generator + + +class BaseTestCase(unittest.TestCase): + + work_path = os.path.abspath(os.path.dirname(__file__)) + + @classmethod + def setUpClass(cls): + cls.python_nogil = os.path.abspath(os.getenv('PYTHON_NOGIL')) + cls.local_path = os.path.join(cls.work_path, '.pyarmor') + cls.default_output = os.path.join(cls.work_path, 'dist') + + def setUp(self): + shutil.rmtree(self.local_path, ignore_errors=True) + + def tearDown(self): + pass + + def assert_python_ok(self, *args): + kwargs = { + '__isolated': False + } + return script_helper.assert_python_ok(*args, **kwargs) + + def assert_python_failure(self, *args): + kwargs = { + '__isolated': False + } + return script_helper.assert_python_failure(*args, **kwargs) + + def verify_script_nogil_pass(self, script, expected): + from subprocess import check_output + assert os.path.exists(self.python_nogil), self.python_nogil + stdout = check_output([self.python_nogil, script]) + self.assertEqual(expected[1], stdout) + + def build_v9_script(self, script, target): + kwargs = { + '__isolated': False, + '__cwd': self.work_path, + } + args = ['-m', 'pyarmor.cli', 'init', '-e', os.path.abspath(script)] + script_helper.assert_python_ok(*args, **kwargs) + + args = ['-m', 'pyarmor.cli', 'build', '--' + target] + script_helper.assert_python_ok(*args, **kwargs) + return os.path.join(self.default_output, os.path.basename(script)) + + def _test_target(self, target, catalog=None): + for script in self.iter_scripts(catalog): + with self.subTest(script=script): + expected = self.assert_python_ok(script) + obfscript = self.build_v9_script(script, target) + self.verify_script_nogil_pass(obfscript, expected) + + def make_script(self, source, name='foo'): + return script_helper.make_script(self.work_path, name, source) + + def iter_samples(self): + path = os.path.join(self.work_path, 'samples') + with os.scandir(path) as it: + for entry in it: + if entry.name.endswith('.py') and entry.is_file(): + yield entry.path + + def iter_scripts(self, catalog): + if catalog: + for name, source in script_generator(catalog): + yield self.make_script(source, name=name) + else: + yield from self.iter_samples() + + +class UnitTestCases(BaseTestCase): + + def test_mini_mode(self): + self._test_target('mini') + self._test_target('mini', 'modules') + self._test_target('mini', 'functions') + self._test_target('mini', 'methods') + + def test_vmc_mode(self): + self._test_target('vmc') + self._test_target('vmc', 'modules') + self._test_target('vmc', 'functions') + self._test_target('vmc', 'methods') + + def test_ecc_mode(self): + self._test_target('ecc-nogil') + self._test_target('ecc-nogil', 'modules') + self._test_target('ecc-nogil', 'functions') + self._test_target('ecc-nogil', 'methods') + + def test_rft_mode(self): + self._test_target('rft') + self._test_target('rft', 'modules') + self._test_target('rft', 'functions') + self._test_target('rft', 'methods') + + def test_mini_rft_mode(self): + self._test_target('mini-rft') + self._test_target('mini-rft', 'modules') + self._test_target('mini-rft', 'functions') + self._test_target('mini-rft', 'methods') + + def test_vmc_rft_mode(self): + self._test_target('vmc-rft') + self._test_target('vmc-rft', 'modules') + self._test_target('vmc-rft', 'functions') + self._test_target('vmc-rft', 'methods') + + def test_ecc_rft_mode(self): + self._test_target('ecc-nogil-rft') + self._test_target('ecc-nogil-rft', 'modules') + self._test_target('ecc-nogil-rft', 'functions') + self._test_target('ecc-nogil-rft', 'methods') + + +if __name__ == '__main__': + logging.getLogger().addHandler(logging.NullHandler()) + + loader = unittest.TestLoader() + # loader.testMethodPrefix = 'test_nogil' + suite = loader.loadTestsFromTestCase(UnitTestCases) + result = unittest.TextTestRunner(verbosity=2).run(suite) diff --git a/tests.9/script_factory.py b/tests.9/script_factory.py new file mode 100755 index 00000000..999b7b21 --- /dev/null +++ b/tests.9/script_factory.py @@ -0,0 +1,921 @@ +# -*- coding: utf-8 -*- +# +# 生成脚本实例 +# +# 使用方法: +# +# from test.support import script_helper +# from script_factory import script_generator +# +# for name, source in script_generator('functions'): +# script_helper.make_script(source, name=name) +# + +import itertools +import sys + +from string import Template +from textwrap import dedent, indent + + +PYMAJOR, PYMINOR = sys.version_info[:2] + + +INDENT = 4 +"""int: Python 脚本的缩进宽度""" + + +module_tpl = Template('''\ +$body +''') + +function_tpl = Template('''\ +def foo(): + $body +foo() +''') + +class_tpl = Template('''\ +class Foo(): + $body +Foo() +''') + +method_tpl = Template('''\ +class Foo(): + def run(self): + $body +Foo().run() +''') + +nest_function_tpl = Template('''\ +def foo(): + def nest(): + $body + nest() +foo() +''') + +nest_class_tpl = Template('''\ +$body +''') + +nest_class_method_tpl = Template('''\ +$body +''') + + +tpl_catalog = { + 'modules': (module_tpl, 0), + 'functions': (function_tpl, 1), + 'classes': (class_tpl, 1), + 'methods': (method_tpl, 2), + 'nest_functions': (nest_function_tpl, 2), +} + + +expr_call_materials = [ + ('call_1', dedent('''\ + print('ok') + ''')), + ('call_2', dedent('''\ + print('ok', end='$') + ''')), + ('call_3', dedent('''\ + print(*('ok', 'yes')) + ''')), + ('call_3_1', dedent('''\ + print(*['ok2', 'yes2']) + ''')), + ('call_4', dedent('''\ + print(*('ok', 'yes'), end='$') + ''')), + ('call_5', dedent('''\ + args = ('ok', 'yes') + kwargs = {'end': '$'} + print(*args, **kwargs) + ''')), + ('call_6', dedent('''\ + args = ('ok', 'yes') + kwargs = {'end': '$'} + print('abc', *args, '123', sep='\t', **kwargs) + ''')), +] + +assign_materials = [ + ('assign_1', dedent('''\ + a = 1 + print(a) + ''')), + ('assign_2', dedent('''\ + x = [0, 1] + x[1] = 2 + print(x) + ''')), + ('assign_3', dedent('''\ + x = [0, 1] + x[1:1] = 4, 5, 6 + print(x) + ''')), + ('assign_3_1', dedent('''\ + x = [0, 1] + x[:] = 4, 5, 6 + print(x) + ''')), + ('assign_3_2', dedent('''\ + x = [0, 1] + x[:1] = 4, 5, 6 + print(x) + ''')), + ('assign_3_3', dedent('''\ + x = [0, 1] + x[1:] = 4, 5, 6 + print(x) + ''')), + ('assign_4', dedent('''\ + x = {} + x['k'] = 2 + print(x) + ''')), + ('assign_5', dedent('''\ + class C: pass + a = C() + a.x = 1 + print(a.x) + ''')), + ('assign_6', dedent('''\ + a, b, c = 1, 2, 3 + print(a, b, c) + ''')), + ('assign_7', dedent('''\ + a, (b, c), d = 1, (2, 3), 5 + print(a, b, c, d) + ''')), + ('assign_7_1', dedent('''\ + a, [b, c], d = 1, (2, 3), 5 + print(a, b, c, d) + ''')), + ('assign_7_2', dedent('''\ + [b, c], = (2, 3), + print(b, c) + ''')), + ('assign_8', dedent('''\ + a, b, *c = 1, (2, 3), 5 + print(a, b, c) + ''')), + ('assign_8_1', dedent('''\ + a, *b, c, d = 1, 2, 3, 5, 6, 7 + print(a, b, c, d) + ''')), + ('assign_8_2', dedent('''\ + *a, (b, c), d = 1, (2, 3), 5 + print(a, b, c, d) + ''')), + ('assign_8_3', dedent('''\ + *a, = 1, (2, 3), 5, 6, 7 + print(a) + ''')), + ('assign_9', dedent('''\ + a, (b, c), *d = 1, (2, 3), 5, 6, 7 + print(a, b, c, d) + ''')), + ('assign_10', dedent('''\ + x = [0, 1] + i = 0 + i, x[i] = 1, 2 + print(i, x) + ''')) +] + +augassign_materials = [ + ('augassign_1', dedent('''\ + a = 1 + a += 2 + a -= 1 + a *= 3 + print(a) + ''')), + ('augassign_2', dedent('''\ + x = [0, 1] + x[1] += 2 + x[1] /= 3 + print(x) + ''')), + ('augassign_3', dedent('''\ + x = {} + x['k'] = 2 + x['k'] += 2 + print(x) + ''')), + ('augassign_4', dedent('''\ + class C: pass + a = C() + a.x = 1 + a.x += 1 + print(a.x) + ''')), +] + +annassign_materials = [ + ('annassign_1', dedent('''\ + a: int = 1 + print(a) + ''')), + ('annassign_2', dedent('''\ + (a): int = 1 + print(a) + ''')), + ('annassign_3', dedent('''\ + c:int + ''')), + ('annassign_4', dedent('''\ + x = [0, 1] + x[1]:int + print(x) + ''')), + ('annassign_5', dedent('''\ + class C: pass + a = C() + a.x:int = 2 + print(a.x) + ''')), +] + +delete_materials = [ + ('delete_1', dedent('''\ + a = 1 + del a + print('a: ', 'a' in locals()) + ''')), + ('delete_2', dedent('''\ + x = [0, 1, 2] + del x[1] + print(x) + ''')), + ('delete_2_1', dedent('''\ + x = [0, 1, 2] + del x[:] + print(x) + ''')), + ('delete_3', dedent('''\ + class C: pass + a = C() + a.x = 1 + print(a.x) + ''')), +] + +for_materials = [ + ('for_1', dedent('''\ + for i in (1, 2, 3): + print(i) + ''')), + ('for_2', dedent('''\ + for i, j in zip((1, 2, 3), (4, 5, 6)): + print(i + j) + ''')), + ('for_3', dedent('''\ + for i in (1, 2, 3): + print(i) + if i > 2: + break + ''')), + ('for_4', dedent('''\ + for i in (1, 2, 3): + if i > 2: + continue + print(i) + ''')), + ('for_5', dedent('''\ + for i in (1, 2, 3): + print(i) + else: + print('else ok') + ''')), + ('for_6', dedent('''\ + for i in (1, 2, 3): + print(i) + break + else: + print('else ok') + ''')), + ('for_7', dedent('''\ + for i in (1, 2, 3): + print(i) + break + else: + print('else ok') + ''')), +] + +while_materials = [ + ('while_1', dedent('''\ + i = 0 + while i < 3: + print(i) + i += 1 + ''')), + ('while_2', dedent('''\ + i = 0 + while i < 3: + print(i) + i += 1 + else: + print('else', i) + ''')), + ('while_3', dedent('''\ + i = 0 + while i < 5: + print(i) + if i > 3: + break + i += 1 + ''')), + ('while_4', dedent('''\ + i = 0 + while i < 5: + i += 1 + if i > 3: + continue + print(i) + ''')), + ('while_5', dedent('''\ + i = 0 + while i < 5: + i += 1 + if i > 3: + continue + print(i) + else: + print('else ok') + ''')), + ('while_6', dedent('''\ + i = 0 + while i < 5: + i += 1 + if i > 3: + break + print(i) + else: + print('else ok', i) + ''')), +] + +if_materials = [ + ('if_1', dedent('''\ + if 1: + print('if ok') + ''')), + ('if_2', dedent('''\ + if 0: + print('if ok') + else: + print('else ok') + ''')), +] + +listcomp_materials = [ + ('listcomp_1', dedent('''\ + a = [x for x in range(10)] + print(a) + ''')), + ('listcomp_2', dedent('''\ + a = [x for x in range(10) if x > 2] + print(a) + ''')), + ('listcomp_2_1', dedent('''\ + a = [x for x in range(10) if x < 6] + print(a) + ''')), + ('listcomp_3', dedent('''\ + a = [x for x in range(10) if x > 2 if x % 3 == 0] + print(a) + ''')), + ('listcomp_4', dedent('''\ + a = [(x, y) for x in range(3) for y in range(3)] + print(a) + ''')), + ('listcomp_5', dedent('''\ + a = [(x, y) for x in range(10) if x > 2 for y in range(10)] + print(a) + ''')), + ('listcomp_6', dedent('''\ + a = [(x, y) for x in range(10) for y in range(10) if y > x] + print(a) + ''')), + ('listcomp_7', dedent('''\ + a = [(x, y) for x in range(10) if x > 2 for y in range(10) if y > x] + print(a) + ''')), +] + +setcomp_materials = [ + ('setcomp_1', dedent('''\ + a = {x for x in range(10)} + print(a) + ''')), + ('setcomp_2', dedent('''\ + a = {x for x in range(10) if x > 2} + print(a) + ''')), + ('setcomp_2_1', dedent('''\ + a = {x for x in range(10) if x < 6} + print(a) + ''')), + ('setcomp_3', dedent('''\ + a = {x for x in range(10) if x > 2 if x % 3 == 0} + print(a) + ''')), + ('setcomp_4', dedent('''\ + a = {(x, y) for x in range(3) for y in range(3)} + print(a) + ''')), + ('setcomp_5', dedent('''\ + a = {(x, y) for x in range(10) if x > 2 for y in range(10)} + print(a) + ''')), + ('setcomp_6', dedent('''\ + a = {(x, y) for x in range(10) for y in range(10) if y > x} + print(a) + ''')), + ('setcomp_7', dedent('''\ + a = {(x, y) for x in range(10) if x > 2 for y in range(10) if y > x} + print(a) + ''')), +] + +gencomp_materials = [ + ('gencomp_1', dedent('''\ + a = (x for x in range(10)) + print(list(a)) + ''')), + ('gencomp_2', dedent('''\ + a = (x for x in range(10) if x > 2) + print(list(a)) + ''')), + ('gencomp_2_1', dedent('''\ + a = (x for x in range(10) if x < 6) + print(list(a)) + ''')), + ('gencomp_3', dedent('''\ + a = (x for x in range(10) if x > 2 if x % 3 == 0) + print(list(a)) + ''')), + ('gencomp_4', dedent('''\ + a = ((x, y) for x in range(3) for y in range(3)) + print(list(a)) + ''')), + ('gencomp_5', dedent('''\ + a = ((x, y) for x in range(10) if x > 2 for y in range(10)) + print(list(a)) + ''')), + ('gencomp_6', dedent('''\ + a = ((x, y) for x in range(10) for y in range(10) if y > x) + print(list(a)) + ''')), + ('gencomp_7', dedent('''\ + a = ((x, y) for x in range(10) if x > 2 for y in range(10) if y > x) + print(list(a)) + ''')), +] + +dictcomp_materials = [ + ('dictcomp_1', dedent('''\ + a = {x: x for x in range(10)} + print(a) + ''')), + ('dictcomp_2', dedent('''\ + a = {x: x for x in range(10) if x > 2} + print(a) + ''')), + ('dictcomp_2_1', dedent('''\ + a = {x: x for x in range(10) if x < 6} + print(a) + ''')), + ('dictcomp_3', dedent('''\ + a = {x: x for x in range(10) if x > 2 if x % 3 == 0} + print(a) + ''')), + ('dictcomp_4', dedent('''\ + a = {x: y for x in range(3) for y in range(3)} + print(a) + ''')), + ('dictcomp_5', dedent('''\ + a = {x: y for x in range(10) if x > 2 for y in range(10)} + print(a) + ''')), + ('dictcomp_6', dedent('''\ + a = {x: y for x in range(10) for y in range(10) if y > x} + print(a) + ''')), + ('dictcomp_7', dedent('''\ + a = {x: y for x in range(10) if x > 2 for y in range(10) if y > x} + print(a) + ''')), +] + +namedexpr_materials = [ + ('namedexpr_1', dedent('''\ + (a := 2) + print(a) + ''')), + ('namedexpr_2', dedent('''\ + x = (a := 2) + print(a, x) + ''')), +] + +boolop_materials = [ + ('boolop_1', dedent('''\ + a = 0 or 1 + print(a) + ''')), + ('boolop_2', dedent('''\ + a = 0 and 1 + print(a) + ''')), + ('boolop_3', dedent('''\ + a = 0 or False or True + print(a) + ''')), + ('boolop_4', dedent('''\ + a = 1 and True and 'abc' + print(a) + ''')), +] + +binop_materials = [ + ('binop_1', dedent('''\ + a = 1 + b = 2 + x = a + b + 5 + int('2') + print(x) + print(a * b) + print( b / a) + print(a * b * 5) + print(10 / a / b) + print(17 % 5) + print(7 >> 2) + print(7 << 2) + print(2 ** 5) + print(2 ^ 10) + print(2 | 9) + print(2 & 9) + ''')), +] + +unaryop_materials = [ + ('unaryop_1', dedent('''\ + a = 1 + print(not a) + print(-a) + print(+a) + print(~a) + ''')), +] + +lambda_materials = [ + ('lambda_1', dedent('''\ + a = lambda x: (x + 2) * 3 + print(a(3)) + ''')), +] + +ifexp_materials = [ + ('ifexp_1', dedent('''\ + x = 1 + a = 'ok' if x > 0 else 'yes, else' + print(a) + ''')), + ('ifexp_2', dedent('''\ + x = 1 + a = 'ok' if x < 0 else 'yes else' + print(a) + ''')), +] + +dict_materials = [ + ('dict_1', dedent('''\ + x = {'a': 1, 'b': 2} + print(x) + ''')), + ('dict_2', dedent('''\ + f = {'c': 3} + x = {'a': 1, **f, 'b': 2} + print(x) + ''')), + ('dict_2', dedent('''\ + f = {'c': 3} + g = {'d': 3} + x = {'a': 1, **f, 'b': 2, **g} + print(x) + ''')), +] + +list_materials = [ + ('list_1', dedent('''\ + c = 5 + a = [1, 2, c] + print(a) + ''')), + ('list_2', dedent('''\ + a = [1, 2] + x = [3, 4, *a] + print(x) + ''')), + ('list_3', dedent('''\ + a = [1, 2] + x = [*a, 3, 4, *a, 5, 6] + print(x) + ''')), +] + +tuple_materials = [ + ('tuple_1', dedent('''\ + c = 5 + a = 1, 2, c + print(a) + ''')), + ('tuple_2', dedent('''\ + a = [1, 2] + x = 3, 4, *a + print(x) + ''')), + ('tuple_2_1', dedent('''\ + a = 1, 2 + x = 3, 4, *a + print(x) + ''')), + ('tuple_3', dedent('''\ + a = 1, 2 + x = *a, 3, 4, *a, 5, 6 + print(x) + ''')), +] + +set_materials = [ + ('set_1', dedent('''\ + c = {5} + a = {1, 2} + print(a, c) + ''')), + ('set_2', dedent('''\ + a = [1, 2] + x = {1, 3, 4, *a} + print(x) + ''')), + ('set_2_1', dedent('''\ + a = 1, 2 + x = {3, 4, *a} + print(x) + ''')), + ('set_2_2', dedent('''\ + a = {1, 2} + x = {3, 4, *a} + print(x) + ''')), + ('set_3', dedent('''\ + a = 1, 2 + x = {*a, 3, 4, *a, 5, 6} + print(x) + ''')), +] + +compare_materials = [ + ('compare_1', dedent('''\ + x = 1 > 2 + print(x) + ''')), + ('compare_1_1', dedent('''\ + x = 2 >= 1 + print(x) + ''')), + ('compare_1_2', dedent('''\ + x = 1 < 2 + print(x) + ''')), + ('compare_1_3', dedent('''\ + x = 1 <= 2 + print(x) + ''')), + ('compare_1_4', dedent('''\ + x = (1 < 2) is True + print(x) + ''')), + ('compare_1_5', dedent('''\ + x = (1 < 2) is not True + print(x) + ''')), + ('compare_1_6', dedent('''\ + x = 2 in (1, 2, 3) + print(x) + ''')), + ('compare_1_7', dedent('''\ + x = 2 not in (1, 2, 3) + print(x) + ''')), + ('compare_2', dedent('''\ + x = 5 > 4 > 3 + print(x) + ''')), + ('compare_3', dedent('''\ + x = (5 > 4) > 3 + print(x) + ''')), +] + +joinedstr_materials = [ + ('joinedstr_1', dedent('''\ + x = f"" + print(x) + ''')), + ('joinedstr_1_1', dedent('''\ + x = f"ok" + print(x) + ''')), + ('joinedstr_1_2', dedent('''\ + a = "ok" + x = f"{a}" + print(x) + ''')), + ('joinedstr_2', dedent('''\ + a = 5.82892 + x = f"sin({a}) is {float(a):.3}" + print(x) + ''')), + ('joinedstr_3', dedent('''\ + a = 60.8999 + b = 'ok' + x = f"{a}{b}sin({a}) is {a:.3}" + print(x) + ''')), +] + +try_materials = [ + ('try_1', dedent('''\ + try: + 1 / 0 + except Exception: + print('try ok') + ''')), + ('try_2', dedent('''\ + try: + 1 / 0 + except Exception as e: + print('try ok', e) + ''')), +] + +with_materials = [ + ('with_1', dedent('''\ + class cm (object): + def __enter__(self): + pass + def __exit__(self, type, value, traceback): + pass + with cm(): + print('with ok') + ''')), +] + +yield_materials = [ + ('yield_1', dedent('''\ + def gen(): + for i in range(10): + yield i + print(list(gen())) + ''')), + ('yield_2', dedent('''\ + def gen(): + a = 3 + b = 4 + c = 5 + yield from (a, b, c) + print(list(gen())) + ''')), +] + +match_materials = [ + ('match_1', dedent('''\ + flag = False + match (100, 200): + case (100, 300): # Mismatch: 200 != 300 + print('Case 1') + case (100, 200) if flag: # Successful match, but guard fails + print('Case 2') + case (100, y): # Matches and binds y to 200 + print(f'Case 3, y: {y}') + case _: # Pattern not attempted + print('Case 4, I match anything!') + ''')), +] + +cell_materials = [ + ('cell_1', dedent('''\ + def test_cell_main(): + a = 12 + def test_cell(b): + print('cell is', a) + print('arg is', b) + test_cell('9') + test_cell_main() + ''')), +] + + +script_materials = itertools.chain( + expr_call_materials, + assign_materials, + augassign_materials, + annassign_materials, + delete_materials, + for_materials, + while_materials, + if_materials, + listcomp_materials, + setcomp_materials, + gencomp_materials, + dictcomp_materials, + namedexpr_materials, + boolop_materials, + binop_materials, + unaryop_materials, + lambda_materials, + ifexp_materials, + dict_materials, + list_materials, + tuple_materials, + set_materials, + compare_materials, + joinedstr_materials, + try_materials, + with_materials, + yield_materials, + cell_materials, +) + +locals_materials = [ +] + +super_materials = [ +] + +samename_materials = [ + ('samename_1', dedent('''\ + def gen(): + a = 3 + b = 4 + c = 5 + return a, b, c + print(gen()) + + def gen(): + i = 1 + return i + print(gen()) + ''')), +] + +immortal_refcnt_materials = [ +] + +async_materials = [ +] + +script_catalog = { + 'locals': locals_materials, + 'super': super_materials, + 'samename': samename_materials, + 'async': async_materials, + 'immortal_refcnt': immortal_refcnt_materials, +} + + +def script_generator(catalog=None): + """生成用于测试各种 Python 功能的脚本 + + 参数: + catalog (str): tpl_catalog 或者 script_catalog 中的分类名称 + + 生成: (name, source) + + name 是脚本名称，没有后缀 `.py` + source 是脚本内容 + + """ + tplinfo = tpl_catalog.get(catalog) + if tplinfo: + tpl, level = tplinfo + if level: + col = level * INDENT + prefix = ' ' * col + for name, source in script_materials: + yield name, tpl.substitute( + body=indent(source, prefix)[col:] if level else source) + # if PYMINOR >= 10: + # for name, source in match_materials: + # yield name, tpl.substitute( + # body=indent(source, prefix)[col:] if level else source) + else: + materials = script_catalog.get(catalog, []) + yield from materials