diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index afcedd7044..2ef731b52f 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -111,7 +111,8 @@ jobs: pip --version - name: Install Antlr tool run: | - pip install antlr4-tools + pip install _scripts/antlr4-tools + pip show antlr4-tools - name: Install JavaScript uses: actions/setup-node@v6.3.0 with: @@ -240,7 +241,8 @@ jobs: pip --version - name: Install Antlr tool run: | - pip install antlr4-tools + pip install _scripts/antlr4-tools + pip show antlr4-tools - name: Install Trash shell: bash run: | diff --git a/_scripts/antlr4-tools/LICENSE b/_scripts/antlr4-tools/LICENSE new file mode 100644 index 0000000000..1b2bb3d849 --- /dev/null +++ b/_scripts/antlr4-tools/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2022 Antlr Project + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/_scripts/antlr4-tools/README.md b/_scripts/antlr4-tools/README.md new file mode 100644 index 0000000000..485078a36a --- /dev/null +++ b/_scripts/antlr4-tools/README.md @@ -0,0 +1,151 @@ +# antlr4-tools + +Tools to run antlr4 w/o needing to install java or antlr4! The only requirement is Python3, which is typically installed on all developer machines on all operating systems. + +## Install + +```bash +$ pip install antlr4-tools +``` + +That creates `antlr4` and `antlr4-parse` executables. On Windows, of course, this doesn't just work. You need to add the `...\local-packages\python38\scripts` dir to your `PATH`, which itself might require a fun reboot or perhaps reinstall of the OS. haha. + +### Windows-specific issues + +On Windows, the `pip` command doesn't just work---you need to add the `...\local-packages\python38\scripts` dir to your `PATH`, which itself might require a fun reboot. If you use WSL on Windows, then the pip install will also properly at the scripts directly (if you run from bash shell). + + +1. Go to the Microsoft Store +2. Search in Microsoft Store for Python +3. Select the newest version of Python (3.10). +4. Click the "Get" button. Store installs python and pip at "c:\Users...\AppData\Local\Microsoft\WindowsApps\python.exe" and "c:\Users...\AppData\Local\Microsoft\WindowsApps\pip.exe", respectively. And, it updates the search path immediately with the install. +5. Open a "cmd" terminal. +6. You can now type "python" and "pip", and "pip install antlr4-tools". 7. Unfortunately, it does not add that to the search path. +7. Update the search path to contain `c:\Users...\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p8\LocalCache\local-packages\Python310\Scripts`. You may need to install MSYS2, then do a `find /c/ -name antlr4.exe 2> /dev/null` and enter that path. +8. Or, you can set up an alias to antlr4.exe on that path. + +The good news is that the ANTLR4 Python tool downloads the ANTLR jar in a standard location, and you don't need to do that manually. It's also possible to go in a browser, go to python.org, and download the python package. But, it's likely you will need to update the path for antlr4.exe as before. + +## First run will install Java and ANTLR + +If needed, `antlr4` will download and install Java 11 and the latest ANTLR jar: + +```bash +$ antlr4 +Downloading antlr4-4.11.1-complete.jar +ANTLR tool needs Java to run; install Java JRE 11 yes/no (default yes)? y +Installed Java in /Users/parrt/.jre/jdk-11.0.15+10-jre; remove that dir to uninstall +ANTLR Parser Generator Version 4.11.1 + -o ___ specify output directory where all output is generated + -lib ___ specify location of grammars, tokens files +... +``` + +To override the version of ANTLR jar used, you can pass a `-v ` argument or set `ANTLR4_TOOLS_ANTLR_VERSION` environment variable: + +```bash +$ antlr4 -v 4.9.3 +ANTLR Parser Generator Version 4.9.3 + -o ___ specify output directory where all output is generated + -lib ___ specify location of grammars, tokens files +... +$ ANTLR4_TOOLS_ANTLR_VERSION=4.10.1 antlr4 +ANTLR Parser Generator Version 4.10.1 + -o ___ specify output directory where all output is generated + -lib ___ specify location of grammars, tokens files +... +``` + +## Running ANTLR tool on grammars + +The `antlr4` command forwards all arguments (besides `-v` mentioned above) to the actual ANTLR tool command: + +```bash +$ antlr4 JSON.g4 +$ ls JSON*.java +JSONBaseListener.java JSONLexer.java JSONListener.java JSONParser.java +$ antlr4 -Dlanguage=Python3 -visitor JSON.g4 +$ ls JSON*.py +JSONLexer.py JSONListener.py JSONParser.py JSONVisitor.py +``` + +## Parsing using interpreter + +The `antlr4-parse` command requires ANTLR 4.11 and above (but any version of ANTLR works for the plain `antlr4` command). It accepts the same `-v` argument or environment variable to override the ANTLR jar version used. (Note: `^D` means control-D and indicates "end of input" on Unix but use `^Z` on Windows.) + +Let's play with a simple grammar: + +``` +grammar Expr; +prog: expr EOF ; +expr: expr ('*'|'/') expr + | expr ('+'|'-') expr + | INT + | '(' expr ')' + ; +NEWLINE : [\r\n]+ -> skip; +INT : [0-9]+ ; +``` + +To parse and get the parse tree in text form, use: + +```bash +$ antlr4-parse Expr.g4 prog -tree +10+20*30 +^D +(prog:1 (expr:2 (expr:3 10) + (expr:1 (expr:3 20) * (expr:3 30))) ) +``` + +Here's how to get the tokens and trace through the parse: + +```bash +$ antlr4-parse Expr.g4 prog -tokens -trace +10+20*30 +[@0,0:1='10',,1:0] +[@1,2:2='+',<'+'>,1:2] +[@2,3:4='20',,1:3] +[@3,5:5='*',<'*'>,1:5] +[@4,6:7='30',,1:6] +[@5,9:8='',,2:0] +enter prog, LT(1)=10 +enter expr, LT(1)=10 +consume [@0,0:1='10',<8>,1:0] rule expr +enter expr, LT(1)=+ +consume [@1,2:2='+',<3>,1:2] rule expr +enter expr, LT(1)=20 +consume [@2,3:4='20',<8>,1:3] rule expr +enter expr, LT(1)=* +consume [@3,5:5='*',<1>,1:5] rule expr +enter expr, LT(1)=30 +consume [@4,6:7='30',<8>,1:6] rule expr +exit expr, LT(1)= +exit expr, LT(1)= +exit expr, LT(1)= +consume [@5,9:8='',<-1>,2:0] rule prog +exit prog, LT(1)= +``` + +Here's how to get a visual tree view: + +```bash +$ antlr4-parse Expr.g4 prog -gui +10+20*30 +``` + +The following will pop up in a Java-based GUI window: + + + +On real grammars, it can be useful to get decision-making profiling info: + +```bash +$ antlr4-parse JavaLexer.g4 JavaParser.g4 compilationUnit -profile dump.csv T.java +$ open /tmp/dump.csv +$ head -5 /tmp/dump.csv +Rule,Invocations,Time (ms),Total k,Max k,Ambiguities,DFA cache miss +compilationUnit:0,1,0.164791,1,1,0,1 +compilationUnit:1,42,1.106583,42,1,0,2 +compilationUnit:2,2,1.73675,2,1,0,2 +compilationUnit:3,1,3.969,1,1,0,1 +``` + diff --git a/_scripts/antlr4-tools/antlr4_tool_runner.py b/_scripts/antlr4-tools/antlr4_tool_runner.py new file mode 100644 index 0000000000..c3942cb41a --- /dev/null +++ b/_scripts/antlr4-tools/antlr4_tool_runner.py @@ -0,0 +1,175 @@ +import os +import sys +import re +import subprocess +import time +from shutil import which +from pathlib import Path +from urllib.request import urlopen +from urllib import error +import json + +import jdk # requires install-jdk package + +mvn_repo: str +homedir: Path + + +def initialize_paths(): + global mvn_repo, homedir + homedir = Path.home() + mvn_repo = os.path.join(homedir, '.m2', 'repository', 'org', 'antlr', 'antlr4') + + +def latest_version(): + try: + with urlopen("https://api.github.com/repos/antlr/antlr4/releases/latest", timeout=10) as response: + s = response.read().decode("UTF-8") + tag = json.loads(s)['tag_name'] + return tag.lstrip('v') + except (error.URLError, error.HTTPError, TimeoutError, KeyError): + pass + print("Could not get latest version number, attempting to fall back to latest downloaded version...") + if not os.path.isdir(mvn_repo): + raise FileNotFoundError( + f"Could not determine the latest ANTLR4 version and no cached versions were found in '{mvn_repo}'" + ) + version_dirs = list(filter(lambda directory: re.match(r"[0-9]+\.[0-9]+\.[0-9]+", directory), os.listdir(mvn_repo))) + version_dirs.sort(reverse=True) + if len(version_dirs) == 0: + raise FileNotFoundError("Could not find a previously downloaded antlr4 jar") + else: + latest_version_dir = version_dirs[0] + print(f"Found version '{latest_version_dir}', this version may be out of date") + return latest_version_dir + + +def antlr4_jar(version): + jar = os.path.join(mvn_repo, version, f'antlr4-{version}-complete.jar') + if not os.path.exists(jar): + return download_antlr4(jar, version) + return jar + + +def download_antlr4(jar, version): + s = None + attempts = 5 + for attempt in range(1, attempts + 1): + try: + with urlopen(f"https://github.com/antlr/website-antlr4/raw/refs/heads/gh-pages/download/antlr-{version}-complete.jar", + timeout=60) as response: + print(f"Downloading antlr-{version}-complete.jar") + os.makedirs(os.path.join(mvn_repo, version), exist_ok=True) + s = response.read() + break # success. + except (error.HTTPError) as e: + if e.code == 404: + print(f"ANTLR version {version} does not exist.") + else: + print(f"HTTPError {e.code} on get antlr-{version}-complete.jar") + break # HTTP errors are permanent; no point retrying + except (error.URLError): + print(f"URLError on get antlr-{version}-complete.jar") + except (TimeoutError): + print(f"TimeoutError on get antlr-{version}-complete.jar") + if attempt < attempts: + print("Retrying...") + time.sleep(30) # small delay before retry + + if s is None: + return None + with open(jar, "wb") as f: + f.write(s) + return jar + + +def find_bin_dir(install_dir): + for root, dirs, files in os.walk(install_dir): + if root.endswith("bin"): + return root + return None + + +def install_jre(java_version='11'): + USER_DIR = os.path.expanduser("~") + JRE_DIR = os.path.join(USER_DIR, ".jre") + if os.path.exists(JRE_DIR): + for f in os.listdir(JRE_DIR): + if f.startswith(f"jdk-{java_version}"): + install_dir = os.path.join(JRE_DIR, f) + bindir = find_bin_dir(install_dir) + java = os.path.join(bindir, 'java') + return java + + r = input(f"ANTLR tool needs Java to run; install Java JRE 11 yes/no (default yes)? ") + if r.strip().lower() not in {'yes', 'y', ''}: + exit(1) + install_dir = jdk.install(java_version, jre=True) + print(f"Installed Java in {install_dir}; remove that dir to uninstall") + bindir = find_bin_dir(install_dir) + if bindir is None: + print(f"Can't find bin/java in {install_dir}; installation failed") + return None + java = os.path.join(bindir, 'java') + return java + + +def install_jre_and_antlr(version): + jar = antlr4_jar(version) + java = which("java") + if java is None: + java = install_jre() + if jar is None or java is None: + exit(1) + CHECK_JRE_VERSION = False + if CHECK_JRE_VERSION: + p = subprocess.Popen([java, '--version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + out, err = p.communicate() + out = out.decode("UTF-8").split('\n')[0] + print(f"Running {out}") + return jar, java + + +def process_args(): + argv = sys.argv[1:] + unparsed_args = [] + version = None + i = 0 + while i < len(argv): + arg = argv[i] + if arg == "-v": + if i + 1 >= len(argv): + print("argument -v: expected one argument") + sys.exit(1) + version = argv[i + 1] + i += 2 + continue + unparsed_args.append(arg) + i += 1 + + return unparsed_args, ( + version or os.environ.get("ANTLR4_TOOLS_ANTLR_VERSION") or latest_version() + ) + + +def run_cli(entrypoint): + initialize_paths() + args, version = process_args() + print(f"Using ANTLR version {version}") + jar, java = install_jre_and_antlr(version) + cp = subprocess.run([java, '-cp', jar, entrypoint] + args) + sys.exit(cp.returncode) + + +def tool(): + """Entry point to run antlr4 tool itself""" + run_cli('org.antlr.v4.Tool') + + +def interp(): + """Entry point to run antlr4 profiling using grammar and input file""" + run_cli('org.antlr.v4.gui.Interpreter') + + +if __name__ == '__main__': + tool() diff --git a/_scripts/antlr4-tools/developer-cert-of-origin.txt b/_scripts/antlr4-tools/developer-cert-of-origin.txt new file mode 100644 index 0000000000..1f5dd11abf --- /dev/null +++ b/_scripts/antlr4-tools/developer-cert-of-origin.txt @@ -0,0 +1,49 @@ +This repo uses the Linux Foundation's Developer +Certificate of Origin, DCO, version 1.1. See either +https://developercertificate.org/ or the text below. + +Each commit requires a "signature", which is simple as +using `-s` (not `-S`) to the git commit command: + +git commit -s -m 'This is my commit message' + +Github's pull request process enforces the sig and gives +instructions on how to fix any commits that lack the sig. +See https://github.com/apps/dco for more info. + +----- https://developercertificate.org/ ------ + +Developer Certificate of Origin +Version 1.1 + +Copyright (C) 2004, 2006 The Linux Foundation and its contributors. + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + + +Developer's Certificate of Origin 1.1 + +By making a contribution to this project, I certify that: + +(a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + +(b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + +(c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + +(d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with + this project or the open source license(s) involved. diff --git a/_scripts/antlr4-tools/images/parse-tree.png b/_scripts/antlr4-tools/images/parse-tree.png new file mode 100644 index 0000000000..7c1e4c5d46 Binary files /dev/null and b/_scripts/antlr4-tools/images/parse-tree.png differ diff --git a/_scripts/antlr4-tools/setup.py b/_scripts/antlr4-tools/setup.py new file mode 100644 index 0000000000..503e97d6d0 --- /dev/null +++ b/_scripts/antlr4-tools/setup.py @@ -0,0 +1,31 @@ +from setuptools import setup + +# To RELEASE: +# +# $ pip install --upgrade build setuptools wheel twine # update tools +# $ rm -rf dist build *.egg-info +# $ python -m build +# $ twine upload dist/* + +v = '0.9.9' + +setup( + name='antlr4-tools', + version=v, + py_modules=['antlr4_tool_runner'], + install_requires=[ + "install-jdk" + ], + url='http://www.antlr.org', + license='MIT', + author='Terence Parr', + author_email='parrt@antlr.org', + entry_points={'console_scripts': [ + 'antlr4=antlr4_tool_runner:tool', + 'antlr4-parse=antlr4_tool_runner:interp' + ] + }, + description='Tools to run ANTLR4 tool and grammar interpreter/profiler', + classifiers=['License :: OSI Approved :: MIT License', + 'Intended Audience :: Developers'] +) diff --git a/abb/readme.md b/abb/readme.md index 6df6dfaa3a..5d0f701f0e 100644 --- a/abb/readme.md +++ b/abb/readme.md @@ -22,3 +22,4 @@ based on a set of ABB Rapid sys-files and the existing kuka-krl antlr file ## Issues Known ambiguity in dataList. +