Skip to content

Commit 430ae28

Browse files
Added: Add terminal support for Bitmaps, Sixel (DCS q) and iTerm Image (OSC 1337)
Support for displaying bitmaps inside the terminal has been added via `TerminalBitmap` which can be created from image `byte[]` or sixel bitmap. The bitmaps are sliced to character cell sized slices. The `TerminalBuffer` stores a map for bitmap number to the `TerminalBitmap` loaded in the terminal. The bitmap number and coordinates are encoded in the `long` `TerminalRow.mStyle` for the `TerminalRow` character of a column by `TerminalBitmap#buildOrThrow()` by getting encoded value from `TextStyle.encodeTerminalBitmap()`. The `TerminalRenderer.render()` then checks during rendering terminal output whether a character at a row/coloumn index is a bitmap instead of text by calling `TextStyle.isTerminalBitmap()`, then draws it using `Canvas.drawBitmap()` instead of `Canvas.drawText()`. Sixel images can be created with Sixel Device Control String command sent via `DCS q s..s ST` or `DCS P1; P2; P3; q s..s ST`. - The `TerminalEmulator` interprets sixel sequences, and sends them to `TerminalBuffer` for constructing a `TerminalSixel`. Once the sixel command has been completely processed, a `TerminalBitmap` is created from the `TerminalSixel`. If an error occurred during processing (like OOM), then remaining sixel command is completely read, but is ignored and no sixel is drawn (done by setting `mTerminalSixel` to `null` so that `TerminalBuffer.sixelReadData()` ignores further commands). - Since a sixel sequence can be very long to render a full image and can have length greater than `TERMINAL_CONTROL_ARGS__MAX_LENGTH` (`16384`), the entire sequence is not stored in the `mTerminalControlArgs` buffer before being processed as it will result in an overflow error, instead as soon as length crosses `TERMINAL_CONTROL_ARGS__MAX_LENGTH / 2` and a complete sixel sub command (`#`, `!`, or `"`) has been received, it is immediately processed, and then further commands are read after emptying buffer. - If "rough" horizontal and vertical size of image is received at start of sixel data string with a `Raster Attributes` command, like done by `img2sixel` command, then sixel commands args buffer capacity (`mTerminalControlArgs`) is increased and sixel bitmap in `TerminalSixel` is resized at start, instead of having to keep resizing buffer/bitmap as more sixel data is received, which has a performance hit due to memory reallocations and copying. - The `4` (sixel) value has been added to `CSI` `Primary Device Attributes` terminal response. The `img2sixel` command can be used to display sixel images after installing with `libsixel` package with `pkg install libsixel`, like with `img2sixel --width=1000px image.jpg`. To manually send an escape sequence, check the `digiater.nl` link below, but it is too cumbersome to create images large enough to be easy viewable in the terminal. See Also: - https://vt100.net/docs/vt3xx-gp/chapter14.html - https://en.wikipedia.org/wiki/Sixel - https://www.digiater.nl/openvms/decus/vax90b1/krypton-nasa/all-about-sixels.text iTerm images can be created with `1337` Operating System Control command. - Both `File=` and `MultipartFile=` (chunk based) protocols are supported. The `inline` parameter should be `1` to display inline images in the terminal. Downloading images to Downloads folder with the value `0` will be ignored as that is not supported. - The escape sequences/image data cannot be greater than `TERMINAL_CONTROL_ARGS__MAX_LENGTH` (`16384`) bytes if sent via (`File=`) protocol, otherwise it will be ignored with an overflow error. For larger images, send images via `MultipartFile=` protocol in chunks with `FilePart=`, the `imgcat` utility uses that with 200-byte chunks if `--legacy` flag is not passed. - The `TerminalEmulator` interprets iTerm images sequences and creates an `ITermImage` to process parameters and store the base64 encoded image sent. Once all the data has been received, which can be over multiple `OSC` commands for `MultipartFile=` protocol, the encoded image is decoded to a `byte[]`, which is then passed to `TerminalBuffer`, which creates a `TerminalBitmap` for the image. The `imgcat` utility can be used for sending images, like with `imgcat --width 1000px image.jpg` (`MultipartFile=`) or `imgcat --width 1000px --legacy image.jpg` (`File=`). To manually send an escape sequence, run `echo -en '\e]1337;File=inline=1;keepAspectRatio=0;width=1000px;:' ; base64 -w 0 ./image.jpg ; echo -e '\e\\'` (`File=`). See Also: - https://iterm2.com/documentation-images.html - https://iterm2.com/utilities/imgcat - https://github.com/gnachman/iTerm2-shell-integration/blob/d1d4012068c3c6761d5676c28ed73e0e2df2b715/utilities/imgcat
1 parent a7f2872 commit 430ae28

9 files changed

Lines changed: 1748 additions & 597 deletions

File tree

Lines changed: 396 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,396 @@
1+
package com.termux.terminal;
2+
3+
import android.util.Base64;
4+
5+
import java.util.Arrays;
6+
7+
/**
8+
* An iTerm image received via `OSC 1337`.
9+
*
10+
* - https://iterm2.com/documentation-images.html
11+
*/
12+
public class ITermImage {
13+
14+
public static final String LOG_TAG = "ITermImage";
15+
16+
17+
18+
/** The {@link Enum} that defines {@link ITermImage} state. */
19+
public enum ImageState {
20+
21+
INIT("init", 0),
22+
ARGUMENTS_READ("arguments_read", 1),
23+
IMAGE_READING("image_reading", 2),
24+
IMAGE_READ("image_read", 3),
25+
IMAGE_DECODED("image_decoded", 4),
26+
FAILED("Failed", 5);
27+
28+
private final String name;
29+
private final int value;
30+
31+
ImageState(final String name, final int value) {
32+
this.name = name;
33+
this.value = value;
34+
}
35+
36+
public String getName() {
37+
return name;
38+
}
39+
40+
public int getValue() {
41+
return value;
42+
}
43+
44+
}
45+
46+
47+
48+
protected final TerminalSessionClient mClient;
49+
50+
protected final boolean mIsMultipart;
51+
52+
protected int mWidth = -1;
53+
protected int mHeight = -1;
54+
55+
protected boolean mInline = false;
56+
57+
protected boolean mPreserveAspectRatio = true;
58+
59+
protected final StringBuilder mEncodedImage = new StringBuilder(/* Initial capacity. */ 4096);
60+
protected byte[] mDecodedImage;
61+
62+
/** The current state of the {@link ImageState}. */
63+
protected ImageState mCurrentState = ImageState.INIT;
64+
/** The previous state of the {@link ImageState}. */
65+
protected ImageState mPreviousState = ImageState.INIT;
66+
67+
68+
69+
protected ITermImage(TerminalSessionClient client, boolean isMultiPart) {
70+
mClient = client;
71+
72+
mIsMultipart = isMultiPart;
73+
}
74+
75+
76+
77+
public TerminalSessionClient getClient() {
78+
return mClient;
79+
}
80+
81+
82+
public boolean isMultipart() {
83+
return mIsMultipart;
84+
}
85+
86+
87+
public int getWidth() {
88+
return mWidth;
89+
}
90+
91+
public int getHeight() {
92+
return mHeight;
93+
}
94+
95+
96+
public boolean isInline() {
97+
return mInline;
98+
}
99+
100+
101+
public boolean shouldPreserveAspectRatio() {
102+
return mPreserveAspectRatio;
103+
}
104+
105+
106+
public String getEncodedImage() {
107+
return mEncodedImage.toString();
108+
}
109+
110+
public byte[] getDecodedImage() {
111+
return mDecodedImage;
112+
}
113+
114+
115+
public synchronized ImageState getCurrentState() {
116+
return mCurrentState;
117+
}
118+
119+
public synchronized ImageState getPreviousState() {
120+
return mPreviousState;
121+
}
122+
123+
124+
protected synchronized boolean setState(ImageState newState) {
125+
// The state transition cannot go back or change if already at `ImageState.IMAGE_DECODED`
126+
if (newState.getValue() < mCurrentState.getValue() || mCurrentState == ImageState.IMAGE_DECODED) {
127+
Logger.logError(mClient, LOG_TAG, "Invalid image state transition from \"" + mCurrentState.getName() + "\" to " + "\"" + newState.getName() + "\"");
128+
return false;
129+
}
130+
131+
// The `ImageState.FAILED` can be set again, like to add more errors, but we don't update
132+
// `mPreviousState` with the `mCurrentState` value if its at `ImageState.FAILED` to
133+
// preserve the last valid state.
134+
if (mCurrentState != ImageState.FAILED)
135+
mPreviousState = mCurrentState;
136+
137+
mCurrentState = newState;
138+
return true;
139+
}
140+
141+
142+
protected synchronized boolean setStateFailed(String error) {
143+
if (error != null) {
144+
Logger.logError(mClient, LOG_TAG, error);
145+
}
146+
return setState(ImageState.FAILED);
147+
}
148+
149+
150+
protected synchronized boolean ensureState(ImageState expectedState) {
151+
return ensureState(expectedState, null);
152+
}
153+
154+
protected synchronized boolean ensureState(ImageState expectedState, String functionName) {
155+
if (mCurrentState != expectedState) {
156+
Logger.logError(mClient, LOG_TAG, "The current image state is \"" + mCurrentState.getName() + "\" but expected \"" + expectedState.getName() + "\"" +
157+
(functionName != null ? " while calling '" + functionName : "'") +
158+
" for " + (!mIsMultipart ? "singlepart" : "multipart") + " image");
159+
return false;
160+
}
161+
return true;
162+
}
163+
164+
165+
public synchronized boolean isArgumentsRead() {
166+
return mCurrentState == ImageState.ARGUMENTS_READ;
167+
}
168+
169+
public synchronized boolean isImageReading() {
170+
return mCurrentState == ImageState.IMAGE_READING;
171+
}
172+
173+
public synchronized boolean isImageRead() {
174+
return mCurrentState == ImageState.IMAGE_READ;
175+
}
176+
177+
public synchronized boolean isImageDecoded() {
178+
return mCurrentState == ImageState.IMAGE_DECODED;
179+
}
180+
181+
182+
183+
public synchronized int readArguments(TerminalEmulator terminalEmulator, StringBuilder oscArgs, int index) {
184+
if (!ensureState(ImageState.INIT, "ImageState.readArguments()")) {
185+
return -1;
186+
}
187+
188+
boolean lastParam = false;
189+
while (index < oscArgs.length()) {
190+
char ch = oscArgs.charAt(index);
191+
// End of optional arguments.
192+
if (ch == ':' && !mIsMultipart) {
193+
break;
194+
} else if (ch == ' ') {
195+
index++;
196+
continue;
197+
}
198+
199+
int keyEndIndex = oscArgs.indexOf("=", index);
200+
if (keyEndIndex == -1) {
201+
setStateFailed("The key for an argument not found after index " + index + " in osc argument string: " + oscArgs);
202+
return -1;
203+
}
204+
String argKey = oscArgs.substring(index, keyEndIndex);
205+
206+
int valueEndIndex = oscArgs.indexOf(";", keyEndIndex);
207+
if (valueEndIndex == -1) {
208+
if (!mIsMultipart) {
209+
// The last key value for `File=` command arguments may end with a colon `:` instead of a semi colon `;`.
210+
valueEndIndex = oscArgs.indexOf(":", keyEndIndex);
211+
if (valueEndIndex == -1) {
212+
setStateFailed("The value for an argument not found after index " + index + " in osc argument string: " + oscArgs);
213+
return -1;
214+
} else {
215+
index = valueEndIndex;
216+
lastParam = true;
217+
}
218+
} else {
219+
// The last key value for `MultipartFile=` command arguments may end without a semi colon `;`.
220+
valueEndIndex = oscArgs.length();
221+
index = valueEndIndex;
222+
}
223+
} else {
224+
index = valueEndIndex + 1;
225+
}
226+
227+
if (valueEndIndex <= keyEndIndex) {
228+
setStateFailed("The argument key end index " + keyEndIndex + " is <= to value end index " + valueEndIndex + " in osc argument string: " + oscArgs);
229+
return -1;
230+
}
231+
232+
String argValue = oscArgs.substring(keyEndIndex + 1, valueEndIndex);
233+
234+
if (argKey.equalsIgnoreCase("inline")) {
235+
mInline = argValue.equals("1");
236+
}
237+
else if (argKey.equalsIgnoreCase("preserveAspectRatio")) {
238+
mPreserveAspectRatio = !argValue.equals("0");
239+
}
240+
else if (argKey.equalsIgnoreCase("width")) {
241+
double factor = terminalEmulator.getCellWidthPixels();
242+
int intValueEndIndex = argValue.length();
243+
if (argValue.endsWith("px")) {
244+
factor = 1;
245+
intValueEndIndex -= 2;
246+
} else if (argValue.endsWith("%")) {
247+
factor = 0.01 * terminalEmulator.getCellWidthPixels() * terminalEmulator.getColumns();
248+
intValueEndIndex -= 1;
249+
}
250+
try {
251+
mWidth = (int) (factor * Integer.parseInt(argValue.substring(0, intValueEndIndex)));
252+
} catch (Exception e) {
253+
}
254+
}
255+
else if (argKey.equalsIgnoreCase("height")) {
256+
double factor = terminalEmulator.getCellHeightPixels();
257+
int intValueEndIndex = argValue.length();
258+
if (argValue.endsWith("px")) {
259+
factor = 1;
260+
intValueEndIndex -= 2;
261+
} else if (argValue.endsWith("%")) {
262+
factor = 0.01 * terminalEmulator.getCellHeightPixels() * terminalEmulator.getRows();
263+
intValueEndIndex -= 1;
264+
}
265+
try {
266+
mHeight = (int) (factor * Integer.parseInt(argValue.substring(0, intValueEndIndex)));
267+
} catch (Exception e) {
268+
}
269+
} else {
270+
// `name` and `size` keys are not supported.
271+
}
272+
273+
if (lastParam) {
274+
break;
275+
}
276+
}
277+
278+
setState(ImageState.ARGUMENTS_READ);
279+
280+
return index;
281+
}
282+
283+
284+
public synchronized boolean readImage(StringBuilder oscArgs, int index) {
285+
if (!mIsMultipart) {
286+
if (!ensureState(ImageState.ARGUMENTS_READ, "ImageState.readImage()")) {
287+
return false;
288+
}
289+
290+
if (index < oscArgs.length()) {
291+
int colonIndex = oscArgs.indexOf(":", index);
292+
if (colonIndex >= 0 && colonIndex + 1 < oscArgs.length()) {
293+
setState(ImageState.IMAGE_READING);
294+
int imageStartIndex = colonIndex + 1;
295+
296+
try {
297+
// Appending can cause an increase in capacity and cause an OOM.
298+
mEncodedImage.append(oscArgs.substring(imageStartIndex));
299+
} catch (Throwable t) {
300+
if (t instanceof OutOfMemoryError) System.gc();
301+
setStateFailed("Collecting singlepart image" + " in osc argument string failed: " + t.getMessage());
302+
return false;
303+
}
304+
305+
setState(ImageState.IMAGE_READ);
306+
return true;
307+
}
308+
}
309+
310+
setStateFailed("Failed to read singlepart image from index " + index + " in osc argument string: " + oscArgs);
311+
return false;
312+
} else {
313+
if (mCurrentState != ImageState.IMAGE_READING &&
314+
!ensureState(ImageState.ARGUMENTS_READ, "ImageState.readImage()")) {
315+
return false;
316+
}
317+
318+
// An empty `FilePart=` command could be received as well, so change state before `if` below.
319+
setState(ImageState.IMAGE_READING);
320+
321+
if (index < oscArgs.length()) {
322+
try {
323+
// Appending can cause an increase in capacity and cause an OOM.
324+
mEncodedImage.append(oscArgs.substring(index));
325+
} catch (Throwable t) {
326+
if (t instanceof OutOfMemoryError) System.gc();
327+
setStateFailed("Collecting multipart image" + " in osc argument string failed: " + t.getMessage());
328+
return false;
329+
}
330+
return true;
331+
}
332+
333+
setStateFailed("Failed to read multipart image" + " in osc argument string: " + oscArgs);
334+
return false;
335+
}
336+
}
337+
338+
public synchronized boolean setMultiPartImageRead() {
339+
if (!mIsMultipart) {
340+
Logger.logError(mClient, LOG_TAG, "Attempting to call 'ImageState.setMultiPartImageRead()' for a singlepart image");
341+
return false;
342+
}
343+
344+
// A `FileEnd` command may have been received without a `FilePart=` command preceding it.
345+
if (!ensureState(ImageState.IMAGE_READING, "ImageState.setMultiPartImageRead()")) {
346+
return false;
347+
}
348+
349+
setState(ImageState.IMAGE_READ);
350+
return true;
351+
}
352+
353+
354+
public synchronized boolean decodeImage() {
355+
if (!ensureState(ImageState.IMAGE_READ, "ImageState.decodeImage()")) {
356+
return false;
357+
}
358+
359+
String encodedImageString = null;
360+
try {
361+
if (mEncodedImage.length() < 1) {
362+
setStateFailed("Cannot decoded an empty image");
363+
return false;
364+
}
365+
366+
while (mEncodedImage.length() % 4 != 0) {
367+
mEncodedImage.append('=');
368+
}
369+
370+
encodedImageString = mEncodedImage.toString();
371+
372+
// Clear original encoded image from memory as it is no longer needed.
373+
mEncodedImage.setLength(0);
374+
mEncodedImage.trimToSize();
375+
376+
mDecodedImage = Base64.decode(encodedImageString, Base64.DEFAULT);
377+
if (mDecodedImage == null || mDecodedImage.length < 2) {
378+
setStateFailed("The decoded image is not valid: " + Arrays.toString(mDecodedImage) + "\nimage: " + encodedImageString);
379+
return false;
380+
}
381+
382+
setState(ImageState.IMAGE_DECODED);
383+
return true;
384+
} catch (Throwable t) {
385+
if (t instanceof OutOfMemoryError) {
386+
Logger.logError(mClient, LOG_TAG, "Failed to decode image: " + t.getMessage());
387+
System.gc();
388+
} else {
389+
Logger.logStackTraceWithMessage(mClient, LOG_TAG, "Failed to decode image: " + encodedImageString, t);
390+
}
391+
setStateFailed(null);
392+
return false;
393+
}
394+
}
395+
396+
}

0 commit comments

Comments
 (0)