Add graphics in terminal support: - Sixel and iTerm2 protocols#2973
Add graphics in terminal support: - Sixel and iTerm2 protocols#2973MatanZ wants to merge 10 commits intotermux:masterfrom
Conversation
|
You are a hero my brother |
|
It works! but gifs are being duplicated on the terminal when expanding and collapsing the keyboard Regular images are being displayed fine
|
This is not a bug. img2sixel displays animations by sending each frame as a different sixel with a command to send the cursor to position (1,1) between them (CSI H). This causes each frame to overwrite the previous one. When you remove the keyboard, the screen gets taller, and termux handles this by scrolling down, so the previous frame which was at position (1,1), is now lower, so it is not covered by the following frames. |
|
oh, so everything's working fine then |
Here's a small guide for anyone else who wants to test this PR
|
|
Has issues with big images. For example when using Test image (7779x4191): https://user-images.githubusercontent.com/107305601/188685258-87eb0704-12c5-4b43-a47a-a1deae99e208.jpg Device config: Pixel 5, Android 13, 8 GiB RAM. |
Image displays fine on termux-monet with Device Config: Xiaomi POCO F1, Android 12L, 6GB RAM
|
|
I believe if MatanZ gets this imported and issue 142 gets solved, then MatanZ gets my $100 bounty on Bountysource for that issue. It will be well worth it!! :-D |
|
Thanks, but anyway there is some proper workaround needed to avoid OOM. Larger heap indeed can solve it for a particular case. Not all devices have lots of memory and part is always used by other apps including system, max heap size also vary between device models. Unfortunately I can't propose the best way to implement that, but I think refusing to display supersized bitmaps is better than crashing all sessions. For example if dimensions are too big, user can get a toast message describing the issue. Command line programs should not be able to crash terminal or cause resource overload by just sending some control sequence (no matter whether it is sixel or something else). |
I don't believe that making the terminal refuse to display oversized images is necessary, since most of command line programs always will be able to crash the application when overloaded or abused. That would only limit how the user should use his terminal, creating a "fake error" that doesn't even exist for some users, like the people who has enough RAM for displaying those images. Those people who can display the images wouldn't be able to. But if you guys decide that's the best thing to do, i propose placing a limit only for devices with less than 4GB RAM. |
Just so I can get more testing from the nice people who test my code:
|
|
Tested with imagemagick, and it's displaying fine on my side |
I hope this solves the problem. I put a try{}catch() block around each bitmap operation that allocates memory. This should be similar to what you suggest - ignoring large images, without hard coding a definition of large. I believe it is better without a toast notification. Usually when a terminal cannot (or does not want to) perform a certain operation it just ignores it, without notifying the user. Maybe a line in the log. |
|
@MatanZ Please use dedicated class for long sixel logic in Trying to catch Checking current memory before even trying to display large image could be done too. https://developer.android.com/topic/performance/graphics/load-bitmap#read-bitmap
This is the way. No toasts. Being done in other places too. |
The way graphics display is implemented, it is not possible to implement kitty protocol. It is possible to implement very simplified ("put image here"), but image and placement management, as well as combining text and images is impossible. |
Refactored some code to two new classes. I don't see anything in TerminalEmulator that belongs in another class, or which may become clearer.
I am not sure this is a good idea. I would have to add assumptions about how much memory is used by the android for various bimap manipulations (decode, scale, resize). I'll settle for trying anything the user asks for, and catching the errors. I did not crash termux with this, with all the above, and various other large images. |
|
Thanks. Will take another look later myself during merging. You can also let |
|
Seems like there is a problem with some GIFs. Notice the artifacts at the left part. Same as #142 (comment)? Gif file doesn't seem to be corrupted. screen-20220908-005128.2.mp4This doesn't happen with another gif: screen-20220908-005944.2.mp4 |
|
I cannot see those videos. And can you please include the actual files used, and the commands that cause the problem? |
|
Perhaps the issue with Termux I can reproduce it when connected over SSH to Termux from Konsole on laptop. But I can't reproduce the issue when animation is played via Screenshot with the problem (notice the block with lines on the left side): You can get the file from https://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/Rotating_earth_%28large%29.gif/274px-Rotating_earth_%28large%29.gif libsixel package version differences:
The difference in a build time configuration. Though maybe OpenSUSE also applies some patches to the package. |
Fixes issue with some GIFs. See my comment at termux/termux-app#2973 (comment) Also enable libcurl support.
|
Commit termux/termux-packages@b72be3c resolves |
Fixes issue with some GIFs. See my comment at termux/termux-app#2973 (comment) Also enable libcurl support.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
If some package doesn't work - open a new issue in https://github.com/termux/termux-packages/issues. |
Sixel coverage:
Conclusion: All applications that uses and/or supports Sixel works properly iTerm2 coverage:
Conclusion: Not all applications that uses and/or supports iTerm2 works properly |
Report InfoUser Action: Crash DetailsCrash Thread: Crash Message: Stacktrace |
|
Hi @MatanZ, thanks for implementing sixels and iTerm images. Sorry, it took this long to look over this, didn't get time (not that I had time right now). I have added my changes in 948ffa4 and 430ae28 primarily, but have kept your old ones in a7f2872 for now after rebasing against Users should thoroughly test the terminal for both sixel and iterm images, and also other text editors, etc in case something got broken. I have done by own testing for the entire protocols, but not a variety of images (primarily tested attached interstellar image). Once testing has been done and @MatanZ reviews the changes, I will merge the pull and make a new Termux app version release for F-Droid users. You can grab APK build for current push from https://github.com/termux/termux-app/actions/runs/24187595506?pr=2973 Commit message for 430ae28 is below. Added: Add terminal support for Bitmaps, Sixel ( Support for displaying bitmaps inside the terminal has been added via Sixel images can be created with Sixel Device Control String command sent via
The To manually send an escape sequence, check the See Also:
iTerm images can be created with
The To manually send an escape sequence, run See Also:
284x177
1920x1200
|
|
finally I can get termux again |
|
If someone who already has Termux installed wants to test this without necessarily having to back up, erase and then later reinstall the regular Termux, they can use this I also built a In order to avoid speculation from anyone who notices that the build I linked also currently contains this other PR, I would like to acknowledge that and also emphasize that the termux-generator links I send are not for replacing or competing with the official Termux as a fork with more PRs merged, so that detail should not be interpreted that way. I send them because I want PRs that "request testers" to be accessible to anyone who only has 1 device and for whom uninstalling and reinstalling the regular Termux constantly while testing PRs is frustrating or inconvenient. |
|
I have tested this in comparison to the 
However, I do seem to have found a sixel image test case that When I use 
however, if I reduce the resolution of 
That seems somewhat reasonable; there has to be a size limit at some point, and maybe the size limit of |
|
I see, I think 2048 is a reasonable limit for sixel. I've tested 
however, If iterm2 is just more efficient than sixel, it's probably best for the size limit of sixel to be smaller than the size limit of iterm2 (as it currently is) |
|
There are some discussions at following links about size limits, I choose 2048x2048 (which uses 16MB memory) as max sixel bitmap dimensions as android devices are generally more stricter with RAM allocation for apps than linux distros and will trigger an OOM, especially with all the other data and processes running in termux. Only issue seems to be that sixel command should still have been read completely without being printed to screen, and no image displayed, just like iterm. contour-terminal/contour#680 |
|
iterm (or possibly imgcat) is around 20 times faster than sixel, I did calculations and rendering same image with iterm was around 7-10ms on average, compared to around 200ms or more for sixel. |
It displays no output for a small range of sizes around 2049-2100, then after that it starts displaying this large text artifact in the terminal, but shows similar messages in 
|
|
Can you post logcat errors for that? |
Ah, a repeat value of 4500 is considered as an invalid sixel sequence and further input is not read, and that gets printed to screen. Will have to think on whether to continue reading or keep as is, probably former as 4500 is a valid integer sent by client, we just dont accept it. Other sixel command limits would need to be looked too. |
I understand supporting I suggest removing the arbitrary 16K limit for both |
Agan, I prefer to avoid unnecessary arbitrary limits. Termux does run on screens wider than 2048 pixels, and programs might want to use all that. Images need not be square, so one might want to draw a 2300x50 header for the page. If you want to reduce the chance of OOM, maybe limit actual area, instead of each dimension separately. Not that there are no problems with supporting large sixels. lsix for example, uses maximum sixel size rather than actual screen width to decide its display width. |
Do you happen to know whether Konsole and other terminal emulators are really fully unlimited, or do they just have a size limit that's really big? |
|
There actually need to be limits, for reporting in XTSMGRAPHICS escape sequence. In foot it is 10000 pixels. In konsole, 16384. |
|
Maybe if the limit were set to something around 8192 in Termux, that would be an acceptable compromise? That is somewhat in between the limit agnostic-apollo wanted, and the limits of other terminals. Estimating based on Agnostic-apollo's previous calculation, I think that would put the maximum memory use somewhere in between 100 and 200 MB, which isn't actually much even on Android. (assuming only attempting to display one large image at a time) |
…age (`OSC 1337`) Support for displaying bitmaps inside the terminal has been added via `TerminalBitmap` which can be created from image `byte[]` or sixel bitmap. The bitmaps are sliced to character cell sized slices. The `TerminalBuffer` stores a map for bitmap number to the `TerminalBitmap` loaded in the terminal. The bitmap number and coordinates are encoded in the `long` `TerminalRow.mStyle` for the `TerminalRow` character of a column by `TerminalBitmap#buildOrThrow()` by getting encoded value from `TextStyle.encodeTerminalBitmap()`. The `TerminalRenderer.render()` then checks during rendering terminal output whether a character at a row/coloumn index is a bitmap instead of text by calling `TextStyle.isTerminalBitmap()`, then draws it using `Canvas.drawBitmap()` instead of `Canvas.drawText()`. Sixel images can be created with Sixel Device Control String command sent via `DCS q s..s ST` or `DCS P1; P2; P3; q s..s ST`. - The `TerminalEmulator` interprets sixel sequences, and sends them to `TerminalBuffer` for constructing a `TerminalSixel`. Once the sixel command has been completely processed, a `TerminalBitmap` is created from the `TerminalSixel`. If an error occurred during processing (like OOM), then remaining sixel command is completely read, but is ignored and no sixel is drawn (done by setting `mTerminalSixel` to `null` so that `TerminalBuffer.sixelReadData()` ignores further commands). - Since a sixel sequence can be very long to render a full image and can have length greater than `TERMINAL_CONTROL_ARGS__MAX_LENGTH` (`16384`), the entire sequence is not stored in the `mTerminalControlArgs` buffer before being processed as it will result in an overflow error, instead as soon as length crosses `TERMINAL_CONTROL_ARGS__MAX_LENGTH / 2` and a complete sixel sub command (`#`, `!`, or `"`) has been received, it is immediately processed, and then further commands are read after emptying buffer. - If "rough" horizontal and vertical size of image is received at start of sixel data string with a `Raster Attributes` command, like done by `img2sixel` command, then sixel commands args buffer capacity (`mTerminalControlArgs`) is increased and sixel bitmap in `TerminalSixel` is resized at start, instead of having to keep resizing buffer/bitmap as more sixel data is received, which has a performance hit due to memory reallocations and copying. - The `4` (sixel) value has been added to `CSI` `Primary Device Attributes` terminal response. The `img2sixel` command can be used to display sixel images after installing with `libsixel` package with `pkg install libsixel`, like with `img2sixel --width=1000px image.jpg`. To manually send an escape sequence, check the `digiater.nl` link below, but it is too cumbersome to create images large enough to be easy viewable in the terminal. See Also: - https://vt100.net/docs/vt3xx-gp/chapter14.html - https://en.wikipedia.org/wiki/Sixel - https://www.digiater.nl/openvms/decus/vax90b1/krypton-nasa/all-about-sixels.text iTerm images can be created with `1337` Operating System Control command. - Both `File=` and `MultipartFile=` (chunk based) protocols are supported. The `inline` parameter should be `1` to display inline images in the terminal. Downloading images to Downloads folder with the value `0` will be ignored as that is not supported. - The escape sequences/image data cannot be greater than `TERMINAL_CONTROL_ARGS__MAX_LENGTH` (`16384`) bytes if sent via (`File=`) protocol, otherwise it will be ignored with an overflow error. For larger images, send images via `MultipartFile=` protocol in chunks with `FilePart=`, the `imgcat` utility uses that with 200-byte chunks if `--legacy` flag is not passed. - The `TerminalEmulator` interprets iTerm images sequences and creates an `ITermImage` to process parameters and store the base64 encoded image sent. Once all the data has been received, which can be over multiple `OSC` commands for `MultipartFile=` protocol, the encoded image is decoded to a `byte[]`, which is then passed to `TerminalBuffer`, which creates a `TerminalBitmap` for the image. The `imgcat` utility can be used for sending images, like with `imgcat --width 1000px image.jpg` (`MultipartFile=`) or `imgcat --width 1000px --legacy image.jpg` (`File=`). To manually send an escape sequence, run `echo -en '\e]1337;File=inline=1;keepAspectRatio=0;width=1000px;:' ; base64 -w 0 ./image.jpg ; echo -e '\e\\'` (`File=`). See Also: - https://iterm2.com/documentation-images.html - https://iterm2.com/utilities/imgcat - https://github.com/gnachman/iTerm2-shell-integration/blob/d1d4012068c3c6761d5676c28ed73e0e2df2b715/utilities/imgcat
The `TerminalBitmap.MAX_BITMAP_SIZE` defines the max size of a Terminal bitmap for its pixels. Each pixel is stored on 4 bytes for a `Bitmap.Config.ARGB_8888` bitmap color config. The value should normally be between `100-200MB` depending on device and Android version. Check the variable docs for more info. The sixel image size cannot be greater than `TerminalBitmap.MAX_BITMAP_SIZE`. The repeat value for sixel Graphics Repeat Introducer command cannot be greater than `TerminalSixel.SIXEL__MAX_REPEAT` (`8192`). The iTerm image data sent for `File=` and `MultipartFile=` protocols cannot be greater than `TerminalBitmap.MAX_BITMAP_SIZE` bytes.
|
@MatanZ and @robertkirkman agreed that the limits were low, I have increased them with 5743c5d. I looked into what limits Android itself was using for different components, and I have increased them to support
This has been fixed too. Repeat value cannot be greater than 8192 now, or rendering will be ignored. You can grab APK build for current push from https://github.com/termux/termux-app/actions/runs/24365026780?pr=2973 |
I have tested this with Samsung Galaxy S8+ SM-G955F, and I can confirm that it is now able to display the large (4500x4500) arrow image! It takes a few seconds to load, but I know that the delay is caused by inefficiency inherent to the sixel protocol, since Konsole also takes a long time to display the same sixel on the same device. Just to stress test this, I have also tested this on Samsung Galaxy S III SPH-L710 (released 2012, 2GB RAM). Here are results of that test: It has no problem displaying the small image, and loads it in under 1 second. 
However, attempting to display the large arrow image (4500x4500) results in the activity freezing for about 7 minutes, 
after that screen, it progresses if I just wait, but after the command completes, it creates a large space where the image should be displayed, without displaying the image itself: 
I think this is reasonable to expect. Since sixel is shown by preexisting implementation Konsole to be intensive and inefficient with large resolution images by nature, very old devices should not be expected to be capable of displaying very large sixel images. |
|
Thanks for testing robert. Yeah, loading large images on older devices is not recommended and is bound to cause issues. Termux isn't responding errors are expected too since bitmaps are created and rendered on main UI thread, high end devices could have issues too with very large images if it takes too long. |
|
If you really wanted to, you could theoretically support large images, even on an older device, but you'd need to implement sixel in the same way as the original hardware terminals. These were devices from the 1980s, with a tiny amount of memory, but they could handle infinitely large images (albeit slowly). It's just that they treated sixel as a streaming protocol, rather than an image protocol, so the content was displayed as it was received, rather than trying to buffer the whole thing into memory, and only rendering it at the end. When you receive a row of sixels that is 4500 pixels wide, and your display is only 800 pixels wide, you just render the first 800 columns and ignore the rest - you don't buffer all 4500 columns, most of which will never be seen. And as new sixel rows are added to the image at the bottom of the screen, you're going to have older parts of the image scrolling off the top. So you don't have to retain those older rows if you're low on memory, in the same way that you don't necessarily keep every row of text in the scrollback. Not suggesting you have to implement it this way though. Just something to consider if you're keen to improve support for low-memory devices. |
But modern terminals including termux have a scrollback history. If entire image is not stored, you cannot scroll back to older rows. Termux supports upto 50000 rows, and defaults to 2000 rows, which is large enough for many many sequential images, a 1920 height image uses only 45lines on a 2856 height display. Older devices also dont have a display resolution high enough that it would matter having to support 4K+ images. Users should just remain within their means. |
I'm not saying you should never store any of the image in the scrollback - just that you can choose to drop parts of an image that have scrolled out of view if you're low on memory. Having part of a large image missing from the scrollback is surely better than not seeing any image at all? But feel free to ignore me. It was just an idea. |
|
Its likely more easily doable to free bitmaps that are at start of scrollback history if an OOM occurs when reading a new bitmap to be added to end of scrollback history, instead of freeing parts of an image. Bitmaps that have been completely scrolled out of scrollback history are already removed, note that scrollback history is larger than the current terminal view of the screen. Your idea is actually good, especially for older terminals, but is more likely to complicate things for termux's case. And your optimization wouldn't work for iTerm images anyways as entire base64 encoded image would need to be read before its converted to a bitmap. If sixels is an issue in future, one could create a new bitmap for each row of a sixel input, and bitmap lines scrolled out could be freed, and older bitmap lines at start of scrollback history could be freed too if an OOM occurs. |
18 participants
This commit implements graphics in the terminal, including two different protocols for placing images:
I ran this for a few days, and it seems to me that some people tried it from #142, and saw no crashes, so I hope it is reasonable safe.
TerminalBuffer for constructing a bitmap.
natural places ($,-,#), rather than collecting all in the buffer.
the style attribute is used to store which bitmap slice is displayed
in place of this character.
drawBitmap, instead of drawText.
Support iTerm2 inline image protocol (OSC 1337):
Small emulator changes:
CSI 14 tto give actual sizeCSI 16 t4(sixel) to device attributesFor those who tried this branch already, I force pushed to it in order to remove changes that may not be acceptable.
The main known issue is that if graphics sequence is interrupted (in specific places), the emulator remains stuck waiting for the sequence to end, thus ignoring all input. This requires terminal reset from the menu.