-
Notifications
You must be signed in to change notification settings - Fork 5
Expand file tree
/
Copy pathgo_dtrace_see_all_network_traffic.html
More file actions
484 lines (467 loc) · 50.2 KB
/
go_dtrace_see_all_network_traffic.html
File metadata and controls
484 lines (467 loc) · 50.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
<!--
This file has been auto-generated by main.rs from a markdown file of the same name.
Do not edit it by hand.
-->
<!DOCTYPE html>
<html>
<head>
<title>See all network traffic in a Go program, even when encrypted and compressed</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link type="application/atom+xml" href="/blog/feed.xml" rel="self">
<link rel="shortcut icon" type="image/ico" href="/blog/favicon.ico">
<link rel="stylesheet" type="text/css" href="main.css">
<link rel="stylesheet" href="https://unpkg.com/@highlightjs/[email protected]/styles/default.min.css">
<script type="module" src="main.js" async></script>
<script type="module" src="search.js" async></script>
</head>
<body>
<div id="banner">
<div id="name">
<img id="me" src="me.jpeg">
<span>Philippe Gaultier</span>
</div>
<input id="search" placeholder="🔎 Search" autocomplete=off>
<ul>
<li> <button id="dark-light-mode">Dark/Light</button> </li>
<li> <a href="/blog/body_of_work.html">Body of work</a> </li>
<li> <a href="/blog/articles-by-tag.html">Tags</a> </li>
<li> <a href="https://github.com/gaultier/resume/raw/master/Philippe_Gaultier_resume_en.pdf">
Resume
</a> </li>
<li> <a href="/blog/feed.xml">
<svg viewBox="0 0 24 24" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M5.5 3.5C4.39543 3.5 3.5 4.39543 3.5 5.5V18.5C3.5 19.6046 4.39543 20.5 5.5 20.5H18.5C19.6046 20.5 20.5 19.6046 20.5 18.5V5.5C20.5 4.39543 19.6046 3.5 18.5 3.5H5.5ZM7 19C8.10457 19 9 18.1046 9 17C9 15.8954 8.10457 15 7 15C5.89543 15 5 15.8954 5 17C5 18.1046 5.89543 19 7 19ZM6.14863 10.5052C6.14863 10.0379 6.52746 9.65906 6.99478 9.65906C7.95949 9.65906 8.91476 9.84908 9.80603 10.2183C10.6973 10.5874 11.5071 11.1285 12.1893 11.8107C12.8715 12.4929 13.4126 13.3027 13.7817 14.194C14.1509 15.0852 14.3409 16.0405 14.3409 17.0052C14.3409 17.4725 13.9621 17.8514 13.4948 17.8514C13.0275 17.8514 12.6486 17.4725 12.6486 17.0052C12.6486 16.2627 12.5024 15.5275 12.2183 14.8416C11.9341 14.1556 11.5177 13.5324 10.9927 13.0073C10.4676 12.4823 9.84437 12.0659 9.15842 11.7817C8.47246 11.4976 7.73726 11.3514 6.99478 11.3514C6.52746 11.3514 6.14863 10.9725 6.14863 10.5052ZM7 5.15385C6.53268 5.15385 6.15385 5.53268 6.15385 6C6.15385 6.46732 6.53268 6.84615 7 6.84615C8.33342 6.84615 9.65379 7.10879 10.8857 7.61907C12.1176 8.12935 13.237 8.87728 14.1799 9.82015C15.1227 10.763 15.8707 11.8824 16.3809 13.1143C16.8912 14.3462 17.1538 15.6666 17.1538 17C17.1538 17.4673 17.5327 17.8462 18 17.8462C18.4673 17.8462 18.8462 17.4673 18.8462 17C18.8462 15.4443 18.5397 13.9039 17.9444 12.4667C17.3491 11.0294 16.4765 9.72352 15.3765 8.6235C14.2765 7.52349 12.9706 6.65091 11.5333 6.05558C10.0961 5.46026 8.55566 5.15385 7 5.15385Z" fill="currentColor"/>
</svg>
</a> </li>
<li> <a href="https://www.linkedin.com/in/philippegaultier/">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" data-supported-dps="24x24" fill="currentColor" width="24" height="24" focusable="false">
<path d="M20.5 2h-17A1.5 1.5 0 002 3.5v17A1.5 1.5 0 003.5 22h17a1.5 1.5 0 001.5-1.5v-17A1.5 1.5 0 0020.5 2zM8 19H5v-9h3zM6.5 8.25A1.75 1.75 0 118.3 6.5a1.78 1.78 0 01-1.8 1.75zM19 19h-3v-4.74c0-1.42-.6-1.93-1.38-1.93A1.74 1.74 0 0013 14.19a.66.66 0 000 .14V19h-3v-9h2.9v1.3a3.11 3.11 0 012.7-1.4c1.55 0 3.36.86 3.36 3.66z"/>
</svg>
</a> </li>
<li> <a href="https://github.com/gaultier">
<svg height="32" aria-hidden="true" viewBox="0 0 24 24" version="1.1" width="32" data-view-component="true" fill="currentColor">
<path d="M12.5.75C6.146.75 1 5.896 1 12.25c0 5.089 3.292 9.387 7.863 10.91.575.101.79-.244.79-.546 0-.273-.014-1.178-.014-2.142-2.889.532-3.636-.704-3.866-1.35-.13-.331-.69-1.352-1.18-1.625-.402-.216-.977-.748-.014-.762.906-.014 1.553.834 1.769 1.179 1.035 1.74 2.688 1.25 3.349.948.1-.747.402-1.25.733-1.538-2.559-.287-5.232-1.279-5.232-5.678 0-1.25.445-2.285 1.178-3.09-.115-.288-.517-1.467.115-3.048 0 0 .963-.302 3.163 1.179.92-.259 1.897-.388 2.875-.388.977 0 1.955.13 2.875.388 2.2-1.495 3.162-1.179 3.162-1.179.633 1.581.23 2.76.115 3.048.733.805 1.179 1.825 1.179 3.09 0 4.413-2.688 5.39-5.247 5.678.417.36.776 1.05.776 2.128 0 1.538-.014 2.774-.014 3.162 0 .302.216.662.79.547C20.709 21.637 24 17.324 24 12.25 24 5.896 18.854.75 12.5.75Z"/>
</svg>
</a> </li>
<li> <a href="https://hachyderm.io/@pg">
<svg width="75" height="79" viewBox="0 0 75 79" xmlns="http://www.w3.org/2000/svg" fill="currentColor">
<path d="M73.8393 17.4898C72.6973 9.00165 65.2994 2.31235 56.5296 1.01614C55.05 0.797115 49.4441 0 36.4582 0H36.3612C23.3717 0 20.585 0.797115 19.1054 1.01614C10.5798 2.27644 2.79399 8.28712 0.904997 16.8758C-0.00358524 21.1056 -0.100549 25.7949 0.0682394 30.0965C0.308852 36.2651 0.355538 42.423 0.91577 48.5665C1.30307 52.6474 1.97872 56.6957 2.93763 60.6812C4.73325 68.042 12.0019 74.1676 19.1233 76.6666C26.7478 79.2728 34.9474 79.7055 42.8039 77.9162C43.6682 77.7151 44.5217 77.4817 45.3645 77.216C47.275 76.6092 49.5123 75.9305 51.1571 74.7385C51.1797 74.7217 51.1982 74.7001 51.2112 74.6753C51.2243 74.6504 51.2316 74.6229 51.2325 74.5948V68.6416C51.2321 68.6154 51.2259 68.5896 51.2142 68.5661C51.2025 68.5426 51.1858 68.522 51.1651 68.5058C51.1444 68.4896 51.1204 68.4783 51.0948 68.4726C51.0692 68.4669 51.0426 68.467 51.0171 68.4729C45.9835 69.675 40.8254 70.2777 35.6502 70.2682C26.7439 70.2682 24.3486 66.042 23.6626 64.2826C23.1113 62.762 22.7612 61.1759 22.6212 59.5646C22.6197 59.5375 22.6247 59.5105 22.6357 59.4857C22.6466 59.4609 22.6633 59.4391 22.6843 59.422C22.7053 59.4048 22.73 59.3929 22.7565 59.3871C22.783 59.3813 22.8104 59.3818 22.8367 59.3886C27.7864 60.5826 32.8604 61.1853 37.9522 61.1839C39.1768 61.1839 40.3978 61.1839 41.6224 61.1516C46.7435 61.008 52.1411 60.7459 57.1796 59.7621C57.3053 59.7369 57.431 59.7154 57.5387 59.6831C65.4861 58.157 73.0493 53.3672 73.8178 41.2381C73.8465 40.7606 73.9184 36.2364 73.9184 35.7409C73.9219 34.0569 74.4606 23.7949 73.8393 17.4898Z"/>
<path d="M61.2484 27.0263V48.114H52.8916V27.6475C52.8916 23.3388 51.096 21.1413 47.4437 21.1413C43.4287 21.1413 41.4177 23.7409 41.4177 28.8755V40.0782H33.1111V28.8755C33.1111 23.7409 31.0965 21.1413 27.0815 21.1413C23.4507 21.1413 21.6371 23.3388 21.6371 27.6475V48.114H13.2839V27.0263C13.2839 22.7176 14.384 19.2946 16.5843 16.7572C18.8539 14.2258 21.8311 12.926 25.5264 12.926C29.8036 12.926 33.0357 14.5705 35.1905 17.8559L37.2698 21.346L39.3527 17.8559C41.5074 14.5705 44.7395 12.926 49.0095 12.926C52.7013 12.926 55.6784 14.2258 57.9553 16.7572C60.1531 19.2922 61.2508 22.7152 61.2484 27.0263Z" fill="#928374" />
<defs>
<linearGradient id="paint0_linear_549_34" x1="37.0692" y1="0" x2="37.0692" y2="79" gradientUnits="userSpaceOnUse">
<stop stop-color="#6364FF"/>
<stop offset="1" stop-color="#563ACC"/>
</linearGradient>
</defs>
</svg>
</a> </li>
<li> <a href="https://bsky.app/profile/pgaultier.bsky.social">
<svg fill="currentColor" viewBox="0 0 64 57" width="32" style="width: 32px; height: 28.5px;"><path d="M13.873 3.805C21.21 9.332 29.103 20.537 32 26.55v15.882c0-.338-.13.044-.41.867-1.512 4.456-7.418 21.847-20.923 7.944-7.111-7.32-3.819-14.64 9.125-16.85-7.405 1.264-15.73-.825-18.014-9.015C1.12 23.022 0 8.51 0 6.55 0-3.268 8.579-.182 13.873 3.805ZM50.127 3.805C42.79 9.332 34.897 20.537 32 26.55v15.882c0-.338.13.044.41.867 1.512 4.456 7.418 21.847 20.923 7.944 7.111-7.32 3.819-14.64-9.125-16.85 7.405 1.264 15.73-.825 18.014-9.015C62.88 23.022 64 8.51 64 6.55c0-9.818-8.578-6.732-13.873-2.745Z"/></svg>
</a> </li>
</ul>
</div>
<div id="search-matches" hidden>
</div>
<div id="pseudo-body">
<div class="article-prelude">
<p><a href="/blog"> ⏴ Back to all articles</a></p>
<p class="publication-date">Published on 2025-09-25. Last modified on 2026-03-04.</p>
</div>
<div class="article-title">
<h1>See all network traffic in a Go program, even when encrypted and compressed</h1>
<div class="tags"> <a href="/blog/articles-by-tag.html#go" class="tag">Go</a> <a href="/blog/articles-by-tag.html#dtrace" class="tag">DTrace</a> </div>
</div>
<details class="toc"><summary>Table of contents</summary>
<ul>
<li>
<a href="#level-1-observe-all-write-read-system-calls">Level 1: Observe all write/read system calls</a>
<ul>
<li>
<a href="#trace-multiple-processes">Trace multiple processes</a>
</li>
<li>
<a href="#trace-all-system-calls-for-networking">Trace all system calls for networking</a>
</li>
</ul>
</li>
<li>
<a href="#level-2-only-observe-network-data">Level 2: Only observe network data</a>
<ul>
<li>
<a href="#alternative-the-tcp-provider">Alternative: the tcp provider</a>
</li>
</ul>
</li>
<li>
<a href="#level-3-see-encrypted-data-in-clear">Level 3: See encrypted data in clear</a>
</li>
<li>
<a href="#level-4-see-compressed-data-in-clear">Level 4: See compressed data in clear</a>
</li>
<li>
<a href="#conclusion">Conclusion</a>
</li>
</ul>
</details>
<p><em>For a gentle introduction to DTrace especially in conjunction with Go, see my past article: <a href="/blog/an_optimization_and_debugging_story_go_dtrace.html">An optimization and debugging story with Go and DTrace</a>.</em></p>
<p>My most common use of DTrace is to observe I/O data received and sent by various programs. This is so valuable and that's why I started using DTrace in the first place!</p>
<p>However, sometimes this data is encrypted and/or compressed which makes simplistic approaches not viable.</p>
<p>I hit this problem when implementing the <a href="https://en.wikipedia.org/wiki/OAuth">OAuth2</a> login flow. If you're not familiar, this allows a user with an account on a 'big' website such as Facebook, Google, etc, known as the 'authorization server', to sign-up/login on a third-party website using their existing account, without having to manage additional credentials. In my particular case, this was 'Login with Amazon' (yes, this exists).</p>
<p>Since there are 3 actors exchanging data back and forth, this is paramount to observe all the data on the wire to understand what's going on (and what's going wrong). The catch is, for security, most of this data is sent over TLS (HTTPS), meaning, encrypted. Thankfully we can use DTrace to still see the data in clear.</p>
<p>Let's see how, one step at a time.</p>
<h2 id="level-1-observe-all-write-read-system-calls">
<a class="title" href="#level-1-observe-all-write-read-system-calls">Level 1: Observe all write/read system calls</a>
<a class="hash-anchor" href="#level-1-observe-all-write-read-system-calls" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>This is the easiest and also what <code>iosnoop</code> or <code>dtruss</code> do.</p>
<p>The simplest form is this:</p>
<pre>
<div class="code-header">
<span>Dtrace</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-dtrace"><span class="line-number"></span><span class="code-hl">#pragma D option strsize=16K</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">syscall::write:entry</span>
<span class="line-number"></span><span class="code-hl">/ pid == $target && arg0 > 0/</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("fd=%d len=%d data=%s\n", arg0, arg2, stringof(copyin(arg1, arg2)));</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">syscall::read:entry</span>
<span class="line-number"></span><span class="code-hl">/ pid == $target /</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> self->read_ptr = arg1;</span>
<span class="line-number"></span><span class="code-hl"> self->read_fd = arg0;</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">syscall::read:return</span>
<span class="line-number"></span><span class="code-hl">/ pid == $target && self->read_ptr!=0 && arg0 > 0 /</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("fd=%d len=%d data=%s\n", self->read_fd, arg0, stringof(copyin(self->read_ptr, arg0)));</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"> self->read_ptr = 0;</span>
<span class="line-number"></span><span class="code-hl"> self->read_fd = 0;</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>We run it like this: <code>sudo dtrace -s myscript.d -p <pid></code> or <code>sudo dtrace -s myscript.d -c ./myprogram</code>.</p>
<p><code>write(2)</code> is the easiest since it is enough to instrument the entry point of the system call. <code>read(2)</code> must be done in two steps, at the entry point we record what is the pointer to the source data, and in the return probe we know how much data was actually read and we can print it.</p>
<p>What's important is to only trace successful reads/writes (<code>/ arg0 > 0 /</code>). Otherwise, we'll try to copy data with a size of <code>-1</code> which will be cast to an unsigned number with the maximum value, and result in a <code>out of scratch space</code> message from DTrace. Which is DTrace's own 'Out Of Memory' case. Fortunately, DTrace has been designed up-front to handle misbehaving D scripts so that they do not impact the stability of the system, so there are not other consequences than our action failing.</p>
<p>Another point: we could be slightly more conservative by also instrumenting the <code>write(2)</code> system call in two steps, because technically, the data that is going to be written, has not yet been paged in by the kernel, and thus could result in a page fault when we try to copy it. The solution is to do it in two steps, same as <code>read(2)</code>, because at the return point, the data has definitely been paged in. I think this is most likely to happen if you write data from the <code>.rodata</code> or <code>.data</code> sections of the executable which have not yet been accessed, but I have found that nearly all real-world programs write dynamic data which has already been faulted in. The <a href="https://illumos.org/books/dtrace/chp-user.html#chp-user">docs</a> mention this fact.</p>
<h3 id="trace-multiple-processes">
<a class="title" href="#trace-multiple-processes">Trace multiple processes</a>
<a class="hash-anchor" href="#trace-multiple-processes" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h3>
<p>Observing system calls system-wide also has the advantage that we can trace multiple processes (by PID or name) in the same script, which is quite nice:</p>
<pre>
<div class="code-header">
<span>Dtrace</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-dtrace"><span class="line-number"></span><span class="code-hl">syscall::write:entry</span>
<span class="line-number"></span><span class="code-hl">/ pid == 123 || pid == 456 || execname == "curl" /</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("fd=%d len=%d data=%s\n", arg0, arg2, stringof(copyin(arg1, arg2)));</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>Then, no need to provide a PID in the DTrace invocation: <code>sudo dtrace -s myscript.d</code>.</p>
<h3 id="trace-all-system-calls-for-networking">
<a class="title" href="#trace-all-system-calls-for-networking">Trace all system calls for networking</a>
<a class="hash-anchor" href="#trace-all-system-calls-for-networking" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h3>
<p>Multiple probes can be grouped with commas when they share the same action, so we can instrument <em>all</em> system calls that do networking in a compact manner. Fortunately, they share the same first few arguments in the same order. I did not list every single one here, this just for illustrative purposes:</p>
<pre>
<div class="code-header">
<span>Dtrace</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-dtrace"><span class="line-number"></span><span class="code-hl">syscall::write:entry, syscall::sendto_nocancel:entry, syscall::sendto:entry </span>
<span class="line-number"></span><span class="code-hl">/ pid == $target && arg0>2 && arg0 > 0/</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("fd=%d len=%d data=%s\n", arg0, arg2, stringof(copyin(arg1, arg2)));</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">syscall::read:entry, syscall::recvfrom_nocancel:entry, syscall::recvfrom:entry </span>
<span class="line-number"></span><span class="code-hl">/ pid == $target /</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> self->read_ptr = arg1;</span>
<span class="line-number"></span><span class="code-hl"> self->read_fd = arg0;</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">syscall::read:return, syscall::recvfrom_nocancel:return, syscall::recvfrom:return </span>
<span class="line-number"></span><span class="code-hl">/ pid == $target && arg0 > 0 && self->read_ptr!=0 /</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("fd=%d len=%d data=%s\n", self->read_fd, arg0, stringof(copyin(self->read_ptr, arg0)));</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"> self->read_ptr = 0;</span>
<span class="line-number"></span><span class="code-hl"> self->read_fd = 0;</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<h2 id="level-2-only-observe-network-data">
<a class="title" href="#level-2-only-observe-network-data">Level 2: Only observe network data</a>
<a class="hash-anchor" href="#level-2-only-observe-network-data" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>A typical program will print things on stderr and stdout, load files from disk, and generally do I/O that's not related to networking, and this will show up in our script.</p>
<p>The approach I have used is to record which file descriptors are real sockets. We can do this by tracing the <code>socket(2)</code> or <code>listen(2)</code> system calls and recording the file descriptor in a global map. Then, we only trace I/O system calls that operate on these file descriptors:</p>
<pre>
<div class="code-header">
<span>Diff</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-diff"><span class="line-number"></span><span class="code-hl">+ syscall::socket:return, syscall::listen:return</span>
<span class="line-number"></span><span class="code-hl">+ / pid == $target && arg0 != -1 /</span>
<span class="line-number"></span><span class="code-hl">+ {</span>
<span class="line-number"></span><span class="code-hl">+ socket_fds[arg0] = 1;</span>
<span class="line-number"></span><span class="code-hl">+ }</span>
<span class="line-number"></span><span class="code-hl">+ </span>
<span class="line-number"></span><span class="code-hl">+ syscall::close:entry</span>
<span class="line-number"></span><span class="code-hl">+ / pid == $target && socket_fds[arg0] != 0 /</span>
<span class="line-number"></span><span class="code-hl">+ {</span>
<span class="line-number"></span><span class="code-hl">+ socket_fds[arg0] = 0;</span>
<span class="line-number"></span><span class="code-hl">+ }</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">syscall::write:entry, syscall::sendto_nocancel:entry, syscall::sendto:entry </span>
<span class="line-number"></span><span class="code-hl">- / pid == $target && arg1 != 0 /</span>
<span class="line-number"></span><span class="code-hl">+ / pid == $target && arg1 != 0 && socket_fds[arg0] != 0 /</span>
<span class="line-number"></span><span class="code-hl">{ </span>
<span class="line-number"></span><span class="code-hl"> // [...]</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">syscall::read:entry, syscall::recvfrom_nocancel:entry, syscall::recvfrom:entry </span>
<span class="line-number"></span><span class="code-hl">- / pid == $target /</span>
<span class="line-number"></span><span class="code-hl">+ / pid == $target && socket_fds[arg0] != 0 /</span>
<span class="line-number"></span><span class="code-hl">{ </span>
<span class="line-number"></span><span class="code-hl"> // [...]</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>When a socket is closed, we need to remove it from our set, since this file descriptor could be used later for a file, etc.</p>
<p>This trick is occasionally useful, but this script has one big requirement: it must start before a socket of interest is opened by the program, otherwise this socket will not be traced. For a web server, if we intend to only trace new connections, that's completely fine, but for a program with long-lived connections that we cannot restart, that approach will not work as is.</p>
<p>In this case, we could first run <code>lsof -i TCP -p <pid</code> to see the file descriptor for these connections and then in the <code>BEGIN {}</code> clause of our D script, add these file descriptors to the <code>socket_fds</code> set manually.</p>
<h3 id="alternative-the-tcp-provider">
<a class="title" href="#alternative-the-tcp-provider">Alternative: the tcp provider</a>
<a class="hash-anchor" href="#alternative-the-tcp-provider" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h3>
<p>The <a href="https://docs.oracle.com/en/operating-systems/solaris/oracle-solaris/11.4/dtrace-guide/tcp-probes.html">tcp</a> provider offers two probes of interest for us: <code>send</code> and <code>receive</code>. With these, we can see what data gets sent over tcp, to/from which address and port, etc. This is quite low-level and seems more designed to help debug the TCP stack, but it can be used.</p>
<p>Let's see an example:</p>
<pre>
<div class="code-header">
<span>Dtrace</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-dtrace"><span class="line-number"></span><span class="code-hl">tcp:::send, tcp:::receive</span>
<span class="line-number"></span><span class="code-hl">/pid==$target/ </span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> tracemem(args[0]->pkt_addr->M_dat.MH_databuf, 288);</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>We see:</p>
<pre>
<div class="code-header">
<span>Plaintext</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-plaintext"><span class="line-number"></span><span class="code-hl">[...]</span>
<span class="line-number"></span><span class="code-hl"> 9 511191 .:send </span>
<span class="line-number"></span><span class="code-hl"> 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef</span>
<span class="line-number"></span><span class="code-hl"> 0: 00 00 00 00 00 00 00 00 11 27 00 00 02 00 00 00 .........'......</span>
<span class="line-number"></span><span class="code-hl"> 10: 60 0c 05 00 00 14 06 ff 20 03 00 e8 77 32 44 00 `....... ...w2D.</span>
<span class="line-number"></span><span class="code-hl"> 20: b1 fa c3 e4 9c bd 36 62 2a 00 14 50 40 01 08 27 ......6b*..P@..'</span>
<span class="line-number"></span><span class="code-hl"> 30: 00 00 00 00 00 00 20 04 f4 68 00 50 41 29 e5 72 ...... ..h.PA).r</span>
<span class="line-number"></span><span class="code-hl"> 40: d9 5e 76 73 80 18 08 10 cc 3a 00 00 01 01 08 0a .^vs.....:......</span>
<span class="line-number"></span><span class="code-hl"> 50: e8 0c ec 49 e5 ac 84 15 47 45 54 20 2f 20 48 54 ...I....GET / HT</span>
<span class="line-number"></span><span class="code-hl"> 60: 54 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 77 77 TP/1.1..Host: ww</span>
<span class="line-number"></span><span class="code-hl"> 70: 77 2e 67 6f 6f 67 6c 65 2e 63 6f 6d 0d 0a 55 73 w.google.com..Us</span>
<span class="line-number"></span><span class="code-hl"> 80: 65 72 2d 41 67 65 6e 74 3a 20 47 6f 2d 68 74 74 er-Agent: Go-htt</span>
<span class="line-number"></span><span class="code-hl"> 90: 70 2d 63 6c 69 65 6e 74 2f 31 2e 31 0d 0a 52 65 p-client/1.1..Re</span>
<span class="line-number"></span><span class="code-hl"> a0: 66 65 72 65 72 3a 20 68 74 74 70 3a 2f 2f 67 6f ferer: http://go</span>
<span class="line-number"></span><span class="code-hl"> b0: 6f 67 6c 65 2e 63 6f 6d 0d 0a 41 63 63 65 70 74 ogle.com..Accept</span>
<span class="line-number"></span><span class="code-hl"> c0: 2d 45 6e 63 6f 64 69 6e 67 3a 20 67 7a 69 70 0d -Encoding: gzip.</span>
<span class="line-number"></span><span class="code-hl"> d0: 0a 0d 0a 0a 0c 1a 07 03 8b 89 b2 8a 89 00 22 01 ..............".</span>
<span class="line-number"></span><span class="code-hl"> e0: 04 1a 00 b0 01 00 00 65 00 00 00 00 00 00 00 00 .......e........</span>
<span class="line-number"></span><span class="code-hl"> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</span>
<span class="line-number"></span><span class="code-hl"> 100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</span>
<span class="line-number"></span><span class="code-hl"> 110: 00 00 00 00 00 00 00 00 00 9b 10 01 00 00 00 00 ................</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">[...]</span>
</code></pre>
<p>The only advantage I can see is that we will only see network data (and not disk I/O, stdout logs, etc). Otherwise, the data is hard to read because the probe fires for each TCP frame and the hexdump contains the TCP header. On top, since this is a kernel probe, wean only see the kernel call stack, not the application call stack.</p>
<p>So this approach is mentioned for completeness, but unless you are a kernel developer working on the TCP stack... This is not so useful.</p>
<h2 id="level-3-see-encrypted-data-in-clear">
<a class="title" href="#level-3-see-encrypted-data-in-clear">Level 3: See encrypted data in clear</a>
<a class="hash-anchor" href="#level-3-see-encrypted-data-in-clear" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>Most websites use TLS nowadays (<code>https://</code>) and will encrypt their data before sending it over the wire. Let's take a simple Go program that makes an HTTP GET request over TLS to demonstrate:</p>
<pre>
<div class="code-header">
<span>Go</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-go"><span class="line-number"></span><span class="code-hl">package main</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">import (</span>
<span class="line-number"></span><span class="code-hl"> "crypto/tls"</span>
<span class="line-number"></span><span class="code-hl"> "net/http"</span>
<span class="line-number"></span><span class="code-hl">)</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">func main() {</span>
<span class="line-number"></span><span class="code-hl"> // Force HTTP 1.</span>
<span class="line-number"></span><span class="code-hl"> http.DefaultClient.Transport = &http.Transport{TLSNextProto: make(map[string]func(authority string, c *tls.Conn) http.RoundTripper)}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"> http.Get("https://google.com")</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p><em>I force HTTP 1 because most people are familiar with it, and the data is text, contrary to HTTP 2 or 3 which use binary. But this is just for readability, the DTrace script works either way. For binary data, <code>tracemem</code> should be used to print a hexdump.</em></p>
<p>When using our previous approach, we only see gibberish, as expected:</p>
<pre>
<div class="code-header">
<span>Plaintext</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-plaintext"><span class="line-number"></span><span class="code-hl">9 176 write:entry size=1502 fd=8: �</span>
<span class="line-number"></span><span class="code-hl">10 175 read:return size=1216 fd=8: ��O��b�%�e�ڡXw2d�q�$���NX�d�P�|�`HY-To�dpK �x�8to�DO���_ibc��y��]�ވe��</span>
<span class="line-number"></span><span class="code-hl"> �c��ҿ��^�L#��Ue </span>
</code></pre>
<p>How can we see the data in clear? Well, we need to find where the data gets encrypted/decrypted in our program and observe the input/output, respectively.</p>
<p>In a typical Go program, the functions of interest are <code>crypto/tls.(*halfConn).decrypt</code> and <code>crypto/tls.(*halfConn).encrypt</code>. Since they are private functions, there is a risk that the Go compiler would inline them which would make them invisible to DTrace. But since they have relatively long and complex bodies, this is unlikely.</p>
<p>Their signatures are:</p>
<ul>
<li><code>func (hc *halfConn) decrypt(record []byte) ([]byte, recordType, error)</code></li>
<li><code>func (hc *halfConn) encrypt(record, payload []byte, rand io.Reader) ([]byte, error)</code></li>
</ul>
<p>By trial and error, we instrument these functions so:</p>
<pre>
<div class="code-header">
<span>Dtrace</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-dtrace"><span class="line-number"></span><span class="code-hl">pid$target::crypto?tls.(?halfConn).decrypt:return</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("%s\n", stringof(copyin(arg1, arg0)));</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">pid$target::crypto?tls.(?halfConn).encrypt:entry</span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("%s\n", stringof(copyin(arg4, arg3)));</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>We need to replace the <code>*</code> character by <code>?</code> since <code>*</code> is interpreted by DTrace as a wildcard (any character, any number of times) in the probe name. <code>?</code> is interpreted as: any character, once.</p>
<p>And boom, we can now see the data in clear:</p>
<pre>
<div class="code-header">
<span>Plaintext</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-plaintext"><span class="line-number"></span><span class="code-hl"> 12 273146 crypto/tls.(*halfConn).encrypt:entry GET / HTTP/1.1</span>
<span class="line-number"></span><span class="code-hl">Host: google.com</span>
<span class="line-number"></span><span class="code-hl">User-Agent: Go-http-client/1.1</span>
<span class="line-number"></span><span class="code-hl">Accept-Encoding: gzip</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"> 1 273145 crypto/tls.(*halfConn).decrypt:return HTTP/1.1 200 OK</span>
<span class="line-number"></span><span class="code-hl">Date: Thu, 25 Sep 2025 15:39:00 GMT</span>
<span class="line-number"></span><span class="code-hl">Expires: -1</span>
<span class="line-number"></span><span class="code-hl">Cache-Control: private, max-age=0</span>
<span class="line-number"></span><span class="code-hl">Content-Type: text/html; charset=ISO-8859-1</span>
<span class="line-number"></span><span class="code-hl">Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-NNo3yQEGH1Cydlkm4FuhSg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp</span>
<span class="line-number"></span><span class="code-hl">Accept-CH: Sec-CH-Prefers-Color-Scheme</span>
<span class="line-number"></span><span class="code-hl">P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."</span>
<span class="line-number"></span><span class="code-hl">Content-Encoding: gzip</span>
<span class="line-number"></span><span class="code-hl">Server: gws</span>
<span class="line-number"></span><span class="code-hl">X-XSS-Protection: 0</span>
<span class="line-number"></span><span class="code-hl">X-Frame-Options: SAMEORIGIN</span>
<span class="line-number"></span><span class="code-hl">Set-Cookie: AEC=AaJma5t2IauygzCrcZIEVudn3SEoGHoVuevRl4vUfxpCR5b6Hnusm3RgLIU; expires=Tue, 24-Mar-2026 15:39:00 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax</span>
<span class="line-number"></span><span class="code-hl">Set-Cookie: __Se</span>
</code></pre>
<h2 id="level-4-see-compressed-data-in-clear">
<a class="title" href="#level-4-see-compressed-data-in-clear">Level 4: See compressed data in clear</a>
<a class="hash-anchor" href="#level-4-see-compressed-data-in-clear" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>For compressed data, the same trick as for encrypted data can be used: find the encode/decode functions and print the input/output, respectively.</p>
<p>Let's take a contrived example where our program sends a random, gzipped string over the network:</p>
<pre>
<div class="code-header">
<span>Go</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-go"><span class="line-number"></span><span class="code-hl">package main</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">import (</span>
<span class="line-number"></span><span class="code-hl"> "compress/gzip"</span>
<span class="line-number"></span><span class="code-hl"> "crypto/rand"</span>
<span class="line-number"></span><span class="code-hl"> "crypto/tls"</span>
<span class="line-number"></span><span class="code-hl"> "net/http"</span>
<span class="line-number"></span><span class="code-hl"> "strings"</span>
<span class="line-number"></span><span class="code-hl">)</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">func main() {</span>
<span class="line-number"></span><span class="code-hl"> // Force HTTP 1.</span>
<span class="line-number"></span><span class="code-hl"> http.DefaultClient.Transport = &http.Transport{TLSNextProto: make(map[string]func(authority string, c *tls.Conn) http.RoundTripper)}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"> w := strings.Builder{}</span>
<span class="line-number"></span><span class="code-hl"> gzipWriter := gzip.NewWriter(&w)</span>
<span class="line-number"></span><span class="code-hl"> msg := "hello " + rand.Text()</span>
<span class="line-number"></span><span class="code-hl"> gzipWriter.Write([]byte(msg))</span>
<span class="line-number"></span><span class="code-hl"> gzipWriter.Close()</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"> http.Post("http://google.com", "application/octet-stream", strings.NewReader(w.String()))</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>If we run our previous D script, we can see the HTTP headers just fine, but the body is gibberish (as expected):</p>
<pre>
<div class="code-header">
<span>Plaintext</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-plaintext"><span class="line-number"></span><span class="code-hl"> 9 176 write:entry action=write pid=46367 execname=go-get size=208 fd=6: POST / HTTP/1.1</span>
<span class="line-number"></span><span class="code-hl">Host: google.com</span>
<span class="line-number"></span><span class="code-hl">User-Agent: Go-http-client/1.1</span>
<span class="line-number"></span><span class="code-hl">Content-Length: 56</span>
<span class="line-number"></span><span class="code-hl">Content-Type: application/octet-stream</span>
<span class="line-number"></span><span class="code-hl">Accept-Encoding: gzip</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">�</span>
</code></pre>
<p>We can trace the method <code>compress/gzip.(*Writer).Write</code> to print its input:</p>
<pre>
<div class="code-header">
<span>Dtrace</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-dtrace"><span class="line-number"></span><span class="code-hl">pid$target::compress?gzip.(?Writer).Write:entry </span>
<span class="line-number"></span><span class="code-hl">{</span>
<span class="line-number"></span><span class="code-hl"> printf("%s\n", stringof(copyin(arg1, arg2)));</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>And we see:</p>
<pre>
<div class="code-header">
<span>Plaintext</span>
<button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
</div>
<code class="language-plaintext"><span class="line-number"></span><span class="code-hl"> 10 530790 compress/gzip.(*Writer).Write:entry hello ZYDEDSRZBY7D65DQHWWVSUOVJ5</span>
</code></pre>
<h2 id="conclusion">
<a class="title" href="#conclusion">Conclusion</a>
<a class="hash-anchor" href="#conclusion" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>All of these approaches can be used in conjunction to see data that has been first compressed, perhaps hashed, then encrypted, etc.</p>
<p>Once again, DTrace shines by its versatility compared to other tools such as Wireshark (can only observe data on the wire, if it's encrypted then tough luck) or <code>strace</code> (can only see system calls). It can programmatically inspect user-space memory, kernel memory, get the stack trace, etc.</p>
<p>Note that some data printed from the <code>read(2)</code> and <code>write(2)</code> system calls will still inevitably appear gibberish because it corresponds to a binary protocol, for example DNS requests.</p>
<p>Finally, let's all note that none of this intricate dance would be necessary, if there were some DTrace static probes judiciously placed in the Go standard library or runtime (wink wink to the Go team).</p>
<p><a href="/blog"> ⏴ Back to all articles</a></p>
<div id="donate">
<em>
<p>If you enjoy what you're reading, you want to support me, and can afford it: <a href="https://paypal.me/philigaultier?country.x=DE&locale.x=en_US">Support me</a>. That allows me to write more cool articles!</p>
<p>
This blog is <a href="https://github.com/gaultier/blog">open-source</a>!
If you find a problem, please open a Github issue.
The content of this blog as well as the code snippets are under the <a href="https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)">BSD-3 License</a> which I also usually use for all my personal projects. It's basically free for every use but you have to mention me as the original author.
</p>
</em>
</div>
</div>
</body>
</html>